You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Adam Roberts <AR...@uk.ibm.com> on 2016/04/15 16:01:09 UTC
BytesToBytes and unaligned memory
Hi, I'm testing Spark 2.0.0 on various architectures and have a question,
are we sure if
core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
really is attempting to use unaligned memory access (for the
BytesToBytesMapOffHeapSuite tests specifically)?
Our JDKs on zSystems for example return false for the
java.nio.Bits.unaligned() method and yet if I skip this check and add
s390x to the supported architectures (for zSystems), all thirteen tests
here pass.
The 13 tests here all fail as we do not pass the unaligned requirement
(but perhaps incorrectly):
core/src/test/java/org/apache/spark/unsafe/map/BytesToBytesMapOffHeapSuite.java
and I know the unaligned checking is at
common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java
Either our JDK's method is returning false incorrectly or this test isn't
using unaligned memory access (so the requirement is invalid), there's no
mention of alignment in the test itself.
Any guidance would be very much appreciated, cheers
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: BytesToBytes and unaligned memory
Posted by Adam Roberts <AR...@uk.ibm.com>.
Ted, yep I'm working from the latest code which includes that unaligned
check, for experimenting I've modified that code to ignore the unaligned
check (just go ahead and say we support it anyway, even though our JDK
returns false: the return value of java.nio.Bits.unaligned()).
My Platform.java for testing contains:
private static final boolean unaligned;
static {
boolean _unaligned;
// use reflection to access unaligned field
try {
System.out.println("Checking unaligned support");
Class<?> bitsClass =
Class.forName("java.nio.Bits", false,
ClassLoader.getSystemClassLoader());
Method unalignedMethod = bitsClass.getDeclaredMethod("unaligned");
unalignedMethod.setAccessible(true);
_unaligned = Boolean.TRUE.equals(unalignedMethod.invoke(null));
System.out.println("Used reflection and _unaligned is: " +
_unaligned);
System.out.println("Setting to true anyway for experimenting");
_unaligned = true;
} catch (Throwable t) {
// We at least know x86 and x64 support unaligned access.
String arch = System.getProperty("os.arch", "");
//noinspection DynamicRegexReplaceableByCompiledPattern
// We don't actually get here since we find the unaligned method OK
and it returns false (I override with true anyway)
// but add s390x incase we somehow fail anyway.
System.out.println("Checking for s390x, os.arch is: " + arch);
_unaligned = arch.matches("^(i[3-6]86|x86(_64)?|x64|s390x|amd64)$");
}
unaligned = _unaligned;
System.out.println("returning: " + unaligned);
}
}
Output is, as you'd expect, "used reflection and _unaligned is false,
setting to true anyway for experimenting", and the tests pass.
No other problems on the platform (pending a different pull request).
Cheers,
From: Ted Yu <yu...@gmail.com>
To: Adam Roberts/UK/IBM@IBMGB
Cc: "dev@spark.apache.org" <de...@spark.apache.org>
Date: 15/04/2016 15:32
Subject: Re: BytesToBytes and unaligned memory
I assume you tested 2.0 with SPARK-12181 .
Related code from Platform.java if java.nio.Bits#unaligned() throws
exception:
// We at least know x86 and x64 support unaligned access.
String arch = System.getProperty("os.arch", "");
//noinspection DynamicRegexReplaceableByCompiledPattern
_unaligned = arch.matches("^(i[3-6]86|x86(_64)?|x64|amd64)$");
Can you give us some detail on how the code runs for JDKs on zSystems ?
Thanks
On Fri, Apr 15, 2016 at 7:01 AM, Adam Roberts <AR...@uk.ibm.com> wrote:
Hi, I'm testing Spark 2.0.0 on various architectures and have a question,
are we sure if
core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
really is attempting to use unaligned memory access (for the
BytesToBytesMapOffHeapSuite tests specifically)?
Our JDKs on zSystems for example return false for the
java.nio.Bits.unaligned() method and yet if I skip this check and add
s390x to the supported architectures (for zSystems), all thirteen tests
here pass.
The 13 tests here all fail as we do not pass the unaligned requirement
(but perhaps incorrectly):
core/src/test/java/org/apache/spark/unsafe/map/BytesToBytesMapOffHeapSuite.java
and I know the unaligned checking is at
common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java
Either our JDK's method is returning false incorrectly or this test isn't
using unaligned memory access (so the requirement is invalid), there's no
mention of alignment in the test itself.
Any guidance would be very much appreciated, cheers
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: BytesToBytes and unaligned memory
Posted by Adam Roberts <AR...@uk.ibm.com>.
Ted, yeah with the forced true value the tests in that suite all pass and
I know they're being executed thanks to prints I've added
Cheers,
From: Ted Yu <yu...@gmail.com>
To: Adam Roberts/UK/IBM@IBMGB
Cc: "dev@spark.apache.org" <de...@spark.apache.org>
Date: 15/04/2016 16:43
Subject: Re: BytesToBytes and unaligned memory
Can you clarify whether BytesToBytesMapOffHeapSuite passed or failed with
the forced true value for unaligned ?
If the test failed, please pastebin the failure(s).
Thanks
On Fri, Apr 15, 2016 at 8:32 AM, Adam Roberts <AR...@uk.ibm.com> wrote:
Ted, yep I'm working from the latest code which includes that unaligned
check, for experimenting I've modified that code to ignore the unaligned
check (just go ahead and say we support it anyway, even though our JDK
returns false: the return value of java.nio.Bits.unaligned()).
My Platform.java for testing contains:
private static final boolean unaligned;
static {
boolean _unaligned;
// use reflection to access unaligned field
try {
System.out.println("Checking unaligned support");
Class<?> bitsClass =
Class.forName("java.nio.Bits", false,
ClassLoader.getSystemClassLoader());
Method unalignedMethod = bitsClass.getDeclaredMethod("unaligned");
unalignedMethod.setAccessible(true);
_unaligned = Boolean.TRUE.equals(unalignedMethod.invoke(null));
System.out.println("Used reflection and _unaligned is: " +
_unaligned);
System.out.println("Setting to true anyway for experimenting");
_unaligned = true;
} catch (Throwable t) {
// We at least know x86 and x64 support unaligned access.
String arch = System.getProperty("os.arch", "");
//noinspection DynamicRegexReplaceableByCompiledPattern
// We don't actually get here since we find the unaligned method OK
and it returns false (I override with true anyway)
// but add s390x incase we somehow fail anyway.
System.out.println("Checking for s390x, os.arch is: " + arch);
_unaligned = arch.matches("^(i[3-6]86|x86(_64)?|x64|s390x|amd64)$");
}
unaligned = _unaligned;
System.out.println("returning: " + unaligned);
}
}
Output is, as you'd expect, "used reflection and _unaligned is false,
setting to true anyway for experimenting", and the tests pass.
No other problems on the platform (pending a different pull request).
Cheers,
From: Ted Yu <yu...@gmail.com>
To: Adam Roberts/UK/IBM@IBMGB
Cc: "dev@spark.apache.org" <de...@spark.apache.org>
Date: 15/04/2016 15:32
Subject: Re: BytesToBytes and unaligned memory
I assume you tested 2.0 with SPARK-12181 .
Related code from Platform.java if java.nio.Bits#unaligned() throws
exception:
// We at least know x86 and x64 support unaligned access.
String arch = System.getProperty("os.arch", "");
//noinspection DynamicRegexReplaceableByCompiledPattern
_unaligned = arch.matches("^(i[3-6]86|x86(_64)?|x64|amd64)$");
Can you give us some detail on how the code runs for JDKs on zSystems ?
Thanks
On Fri, Apr 15, 2016 at 7:01 AM, Adam Roberts <AR...@uk.ibm.com> wrote:
Hi, I'm testing Spark 2.0.0 on various architectures and have a question,
are we sure if
core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
really is attempting to use unaligned memory access (for the
BytesToBytesMapOffHeapSuite tests specifically)?
Our JDKs on zSystems for example return false for the
java.nio.Bits.unaligned() method and yet if I skip this check and add
s390x to the supported architectures (for zSystems), all thirteen tests
here pass.
The 13 tests here all fail as we do not pass the unaligned requirement
(but perhaps incorrectly):
core/src/test/java/org/apache/spark/unsafe/map/BytesToBytesMapOffHeapSuite.java
and I know the unaligned checking is at
common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java
Either our JDK's method is returning false incorrectly or this test isn't
using unaligned memory access (so the requirement is invalid), there's no
mention of alignment in the test itself.
Any guidance would be very much appreciated, cheers
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: BytesToBytes and unaligned memory
Posted by Ted Yu <yu...@gmail.com>.
bq. run the tests claiming to require unaligned memory access on a platform
where unaligned memory access is definitely not supported for
shorts/ints/longs.
That would help us understand interactions on s390x platform better.
On Mon, Apr 18, 2016 at 6:49 AM, Adam Roberts <AR...@uk.ibm.com> wrote:
> Ted, yes with the forced true value all tests pass, we use the unaligned
> check in 15 other suites.
>
> Our java.nio.Bits.unaligned() function checks that the detected os.arch
> value matches a list of known implementations (not including s390x).
>
> We could add it to the known architectures in the catch block but this
> won't make a difference here as because we call unaligned() OK (no
> exception is thrown), we don't reach the architecture checking stage anyway.
>
> I see in org.apache.spark.memory.MemoryManager that unaligned support is
> required for off-heap memory in Tungsten (perhaps incorrectly if no code
> ever exercises it in Spark?). Instead of having a requirement should we
> instead log a warning once that this is likely to lead to slow performance?
> What's the rationale for supporting unaligned memory access: it's my
> understanding that it's typically very slow, are there any design docs or
> perhaps a JIRA where I can learn more?
>
> Will run a simple test case exercising unaligned memory access for Linux
> on Z (without using Spark) and can also run the tests claiming to require
> unaligned memory access on a platform where unaligned memory access is
> definitely not supported for shorts/ints/longs.
>
> if these tests continue to pass then I think the Spark tests don't
> exercise unaligned memory access, cheers
>
>
>
>
>
>
>
> From: Ted Yu <yu...@gmail.com>
> To: Adam Roberts/UK/IBM@IBMGB
> Cc: "dev@spark.apache.org" <de...@spark.apache.org>
> Date: 15/04/2016 17:35
> Subject: Re: BytesToBytes and unaligned memory
> ------------------------------
>
>
>
> I am curious if all Spark unit tests pass with the forced true value for
> unaligned.
> If that is the case, it seems we can add s390x to the known architectures.
>
> It would also give us some more background if you can describe
> how java.nio.Bits#unaligned() is implemented on s390x.
>
> Josh / Andrew / Davies / Ryan are more familiar with related code. It
> would be good to hear what they think.
>
> Thanks
>
> On Fri, Apr 15, 2016 at 8:47 AM, Adam Roberts <*AROBERTS@uk.ibm.com*
> <AR...@uk.ibm.com>> wrote:
> Ted, yeah with the forced true value the tests in that suite all pass and
> I know they're being executed thanks to prints I've added
>
> Cheers,
>
>
>
>
> From: Ted Yu <*yuzhihong@gmail.com* <yu...@gmail.com>>
> To: Adam Roberts/UK/IBM@IBMGB
> Cc: "*dev@spark.apache.org* <de...@spark.apache.org>" <
> *dev@spark.apache.org* <de...@spark.apache.org>>
> Date: 15/04/2016 16:43
> Subject: Re: BytesToBytes and unaligned memory
> ------------------------------
>
>
>
> Can you clarify whether BytesToBytesMapOffHeapSuite passed or failed with
> the forced true value for unaligned ?
>
> If the test failed, please pastebin the failure(s).
>
> Thanks
>
> On Fri, Apr 15, 2016 at 8:32 AM, Adam Roberts <*AROBERTS@uk.ibm.com*
> <AR...@uk.ibm.com>> wrote:
> Ted, yep I'm working from the latest code which includes that unaligned
> check, for experimenting I've modified that code to ignore the unaligned
> check (just go ahead and say we support it anyway, even though our JDK
> returns false: the return value of java.nio.Bits.unaligned()).
>
> My Platform.java for testing contains:
>
> private static final boolean unaligned;
>
> static {
> boolean _unaligned;
> // use reflection to access unaligned field
> try {
> * System.out.println("Checking unaligned support");*
> Class<?> bitsClass =
> Class.forName("java.nio.Bits", false,
> ClassLoader.getSystemClassLoader());
> Method unalignedMethod = bitsClass.getDeclaredMethod("unaligned");
> unalignedMethod.setAccessible(true);
> _unaligned = Boolean.TRUE.equals(unalignedMethod.invoke(null));
> *System.out.println("Used reflection and _unaligned is: " +
> _unaligned);*
> * System.out.println("Setting to true anyway for experimenting");*
> * _unaligned = true;*
> } catch (Throwable t) {
> // We at least know x86 and x64 support unaligned access.
> String arch = System.getProperty("os.arch", "");
> //noinspection DynamicRegexReplaceableByCompiledPattern
> * // We don't actually get here since we find the unaligned method
> OK and it returns false (I override with true anyway)*
> * // but add s390x incase we somehow fail anyway.*
> * System.out.println("Checking for s390x, os.arch is: " + arch);*
> * _unaligned =
> arch.matches("^(i[3-6]86|x86(_64)?|x64|s390x|amd64)$");*
> }
> unaligned = _unaligned;
> * System.out.println("returning: " + unaligned);*
> }
> }
>
> Output is, as you'd expect, "used reflection and _unaligned is false,
> setting to true anyway for experimenting", and the tests pass.
>
> No other problems on the platform (pending a different pull request).
>
> Cheers,
>
>
>
>
>
>
>
> From: Ted Yu <*yuzhihong@gmail.com* <yu...@gmail.com>>
> To: Adam Roberts/UK/IBM@IBMGB
> Cc: "*dev@spark.apache.org* <de...@spark.apache.org>" <
> *dev@spark.apache.org* <de...@spark.apache.org>>
> Date: 15/04/2016 15:32
> Subject: Re: BytesToBytes and unaligned memory
> ------------------------------
>
>
>
>
> I assume you tested 2.0 with SPARK-12181 .
>
> Related code from Platform.java if java.nio.Bits#unaligned() throws
> exception:
>
> // We at least know x86 and x64 support unaligned access.
> String arch = System.getProperty("os.arch", "");
> //noinspection DynamicRegexReplaceableByCompiledPattern
> _unaligned = arch.matches("^(i[3-6]86|x86(_64)?|x64|amd64)$");
>
> Can you give us some detail on how the code runs for JDKs on zSystems ?
>
> Thanks
>
> On Fri, Apr 15, 2016 at 7:01 AM, Adam Roberts <*AROBERTS@uk.ibm.com*
> <AR...@uk.ibm.com>> wrote:
> Hi, I'm testing Spark 2.0.0 on various architectures and have a question,
> are we sure if
> *core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java*
> <https://github.com/apache/spark/blob/96941b12f8b465df21423275f3cd3ade579b4fa1/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java>
> really is attempting to use unaligned memory access (for the
> BytesToBytesMapOffHeapSuite tests specifically)?
>
> Our JDKs on zSystems for example return false for the
> java.nio.Bits.unaligned() method and yet if I skip this check and add s390x
> to the supported architectures (for zSystems), all thirteen tests here
> pass.
>
> The 13 tests here all fail as we do not pass the unaligned requirement
> (but perhaps incorrectly):
>
> *core/src/test/java/org/apache/spark/unsafe/map/BytesToBytesMapOffHeapSuite.java*
> <https://github.com/apache/spark/blob/d6dc12ef0146ae409834c78737c116050961f350/core/src/test/java/org/apache/spark/unsafe/map/BytesToBytesMapOffHeapSuite.java>
> and I know the unaligned checking is at
> *common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java*
> <https://github.com/apache/spark/blob/master/common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java>
>
> Either our JDK's method is returning false incorrectly or this test isn't
> using unaligned memory access (so the requirement is invalid), there's no
> mention of alignment in the test itself.
>
> Any guidance would be very much appreciated, cheers
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
Re: BytesToBytes and unaligned memory
Posted by Adam Roberts <AR...@uk.ibm.com>.
Ted, yes with the forced true value all tests pass, we use the unaligned
check in 15 other suites.
Our java.nio.Bits.unaligned() function checks that the detected os.arch
value matches a list of known implementations (not including s390x).
We could add it to the known architectures in the catch block but this
won't make a difference here as because we call unaligned() OK (no
exception is thrown), we don't reach the architecture checking stage
anyway.
I see in org.apache.spark.memory.MemoryManager that unaligned support is
required for off-heap memory in Tungsten (perhaps incorrectly if no code
ever exercises it in Spark?). Instead of having a requirement should we
instead log a warning once that this is likely to lead to slow
performance? What's the rationale for supporting unaligned memory access:
it's my understanding that it's typically very slow, are there any design
docs or perhaps a JIRA where I can learn more?
Will run a simple test case exercising unaligned memory access for Linux
on Z (without using Spark) and can also run the tests claiming to require
unaligned memory access on a platform where unaligned memory access is
definitely not supported for shorts/ints/longs.
if these tests continue to pass then I think the Spark tests don't
exercise unaligned memory access, cheers
From: Ted Yu <yu...@gmail.com>
To: Adam Roberts/UK/IBM@IBMGB
Cc: "dev@spark.apache.org" <de...@spark.apache.org>
Date: 15/04/2016 17:35
Subject: Re: BytesToBytes and unaligned memory
I am curious if all Spark unit tests pass with the forced true value for
unaligned.
If that is the case, it seems we can add s390x to the known architectures.
It would also give us some more background if you can describe
how java.nio.Bits#unaligned() is implemented on s390x.
Josh / Andrew / Davies / Ryan are more familiar with related code. It
would be good to hear what they think.
Thanks
On Fri, Apr 15, 2016 at 8:47 AM, Adam Roberts <AR...@uk.ibm.com> wrote:
Ted, yeah with the forced true value the tests in that suite all pass and
I know they're being executed thanks to prints I've added
Cheers,
From: Ted Yu <yu...@gmail.com>
To: Adam Roberts/UK/IBM@IBMGB
Cc: "dev@spark.apache.org" <de...@spark.apache.org>
Date: 15/04/2016 16:43
Subject: Re: BytesToBytes and unaligned memory
Can you clarify whether BytesToBytesMapOffHeapSuite passed or failed with
the forced true value for unaligned ?
If the test failed, please pastebin the failure(s).
Thanks
On Fri, Apr 15, 2016 at 8:32 AM, Adam Roberts <AR...@uk.ibm.com> wrote:
Ted, yep I'm working from the latest code which includes that unaligned
check, for experimenting I've modified that code to ignore the unaligned
check (just go ahead and say we support it anyway, even though our JDK
returns false: the return value of java.nio.Bits.unaligned()).
My Platform.java for testing contains:
private static final boolean unaligned;
static {
boolean _unaligned;
// use reflection to access unaligned field
try {
System.out.println("Checking unaligned support");
Class<?> bitsClass =
Class.forName("java.nio.Bits", false,
ClassLoader.getSystemClassLoader());
Method unalignedMethod = bitsClass.getDeclaredMethod("unaligned");
unalignedMethod.setAccessible(true);
_unaligned = Boolean.TRUE.equals(unalignedMethod.invoke(null));
System.out.println("Used reflection and _unaligned is: " +
_unaligned);
System.out.println("Setting to true anyway for experimenting");
_unaligned = true;
} catch (Throwable t) {
// We at least know x86 and x64 support unaligned access.
String arch = System.getProperty("os.arch", "");
//noinspection DynamicRegexReplaceableByCompiledPattern
// We don't actually get here since we find the unaligned method OK
and it returns false (I override with true anyway)
// but add s390x incase we somehow fail anyway.
System.out.println("Checking for s390x, os.arch is: " + arch);
_unaligned = arch.matches("^(i[3-6]86|x86(_64)?|x64|s390x|amd64)$");
}
unaligned = _unaligned;
System.out.println("returning: " + unaligned);
}
}
Output is, as you'd expect, "used reflection and _unaligned is false,
setting to true anyway for experimenting", and the tests pass.
No other problems on the platform (pending a different pull request).
Cheers,
From: Ted Yu <yu...@gmail.com>
To: Adam Roberts/UK/IBM@IBMGB
Cc: "dev@spark.apache.org" <de...@spark.apache.org>
Date: 15/04/2016 15:32
Subject: Re: BytesToBytes and unaligned memory
I assume you tested 2.0 with SPARK-12181 .
Related code from Platform.java if java.nio.Bits#unaligned() throws
exception:
// We at least know x86 and x64 support unaligned access.
String arch = System.getProperty("os.arch", "");
//noinspection DynamicRegexReplaceableByCompiledPattern
_unaligned = arch.matches("^(i[3-6]86|x86(_64)?|x64|amd64)$");
Can you give us some detail on how the code runs for JDKs on zSystems ?
Thanks
On Fri, Apr 15, 2016 at 7:01 AM, Adam Roberts <AR...@uk.ibm.com> wrote:
Hi, I'm testing Spark 2.0.0 on various architectures and have a question,
are we sure if
core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
really is attempting to use unaligned memory access (for the
BytesToBytesMapOffHeapSuite tests specifically)?
Our JDKs on zSystems for example return false for the
java.nio.Bits.unaligned() method and yet if I skip this check and add
s390x to the supported architectures (for zSystems), all thirteen tests
here pass.
The 13 tests here all fail as we do not pass the unaligned requirement
(but perhaps incorrectly):
core/src/test/java/org/apache/spark/unsafe/map/BytesToBytesMapOffHeapSuite.java
and I know the unaligned checking is at
common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java
Either our JDK's method is returning false incorrectly or this test isn't
using unaligned memory access (so the requirement is invalid), there's no
mention of alignment in the test itself.
Any guidance would be very much appreciated, cheers
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: BytesToBytes and unaligned memory
Posted by Ted Yu <yu...@gmail.com>.
I am curious if all Spark unit tests pass with the forced true value for
unaligned.
If that is the case, it seems we can add s390x to the known architectures.
It would also give us some more background if you can describe how
java.nio.Bits#unaligned()
is implemented on s390x.
Josh / Andrew / Davies / Ryan are more familiar with related code. It would
be good to hear what they think.
Thanks
On Fri, Apr 15, 2016 at 8:47 AM, Adam Roberts <AR...@uk.ibm.com> wrote:
> Ted, yeah with the forced true value the tests in that suite all pass and
> I know they're being executed thanks to prints I've added
>
> Cheers,
>
>
>
>
> From: Ted Yu <yu...@gmail.com>
> To: Adam Roberts/UK/IBM@IBMGB
> Cc: "dev@spark.apache.org" <de...@spark.apache.org>
> Date: 15/04/2016 16:43
> Subject: Re: BytesToBytes and unaligned memory
> ------------------------------
>
>
>
> Can you clarify whether BytesToBytesMapOffHeapSuite passed or failed with
> the forced true value for unaligned ?
>
> If the test failed, please pastebin the failure(s).
>
> Thanks
>
> On Fri, Apr 15, 2016 at 8:32 AM, Adam Roberts <*AROBERTS@uk.ibm.com*
> <AR...@uk.ibm.com>> wrote:
> Ted, yep I'm working from the latest code which includes that unaligned
> check, for experimenting I've modified that code to ignore the unaligned
> check (just go ahead and say we support it anyway, even though our JDK
> returns false: the return value of java.nio.Bits.unaligned()).
>
> My Platform.java for testing contains:
>
> private static final boolean unaligned;
>
> static {
> boolean _unaligned;
> // use reflection to access unaligned field
> try {
> * System.out.println("Checking unaligned support");*
> Class<?> bitsClass =
> Class.forName("java.nio.Bits", false,
> ClassLoader.getSystemClassLoader());
> Method unalignedMethod = bitsClass.getDeclaredMethod("unaligned");
> unalignedMethod.setAccessible(true);
> _unaligned = Boolean.TRUE.equals(unalignedMethod.invoke(null));
> *System.out.println("Used reflection and _unaligned is: " +
> _unaligned);*
> * System.out.println("Setting to true anyway for experimenting");*
> * _unaligned = true;*
> } catch (Throwable t) {
> // We at least know x86 and x64 support unaligned access.
> String arch = System.getProperty("os.arch", "");
> //noinspection DynamicRegexReplaceableByCompiledPattern
> * // We don't actually get here since we find the unaligned method
> OK and it returns false (I override with true anyway)*
> * // but add s390x incase we somehow fail anyway.*
> * System.out.println("Checking for s390x, os.arch is: " + arch);*
> * _unaligned =
> arch.matches("^(i[3-6]86|x86(_64)?|x64|s390x|amd64)$");*
> }
> unaligned = _unaligned;
> * System.out.println("returning: " + unaligned);*
> }
> }
>
> Output is, as you'd expect, "used reflection and _unaligned is false,
> setting to true anyway for experimenting", and the tests pass.
>
> No other problems on the platform (pending a different pull request).
>
> Cheers,
>
>
>
>
>
>
>
> From: Ted Yu <*yuzhihong@gmail.com* <yu...@gmail.com>>
> To: Adam Roberts/UK/IBM@IBMGB
> Cc: "*dev@spark.apache.org* <de...@spark.apache.org>" <
> *dev@spark.apache.org* <de...@spark.apache.org>>
> Date: 15/04/2016 15:32
> Subject: Re: BytesToBytes and unaligned memory
> ------------------------------
>
>
>
>
> I assume you tested 2.0 with SPARK-12181 .
>
> Related code from Platform.java if java.nio.Bits#unaligned() throws
> exception:
>
> // We at least know x86 and x64 support unaligned access.
> String arch = System.getProperty("os.arch", "");
> //noinspection DynamicRegexReplaceableByCompiledPattern
> _unaligned = arch.matches("^(i[3-6]86|x86(_64)?|x64|amd64)$");
>
> Can you give us some detail on how the code runs for JDKs on zSystems ?
>
> Thanks
>
> On Fri, Apr 15, 2016 at 7:01 AM, Adam Roberts <*AROBERTS@uk.ibm.com*
> <AR...@uk.ibm.com>> wrote:
> Hi, I'm testing Spark 2.0.0 on various architectures and have a question,
> are we sure if
> *core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java*
> <https://github.com/apache/spark/blob/96941b12f8b465df21423275f3cd3ade579b4fa1/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java>
> really is attempting to use unaligned memory access (for the
> BytesToBytesMapOffHeapSuite tests specifically)?
>
> Our JDKs on zSystems for example return false for the
> java.nio.Bits.unaligned() method and yet if I skip this check and add s390x
> to the supported architectures (for zSystems), all thirteen tests here
> pass.
>
> The 13 tests here all fail as we do not pass the unaligned requirement
> (but perhaps incorrectly):
>
> *core/src/test/java/org/apache/spark/unsafe/map/BytesToBytesMapOffHeapSuite.java*
> <https://github.com/apache/spark/blob/d6dc12ef0146ae409834c78737c116050961f350/core/src/test/java/org/apache/spark/unsafe/map/BytesToBytesMapOffHeapSuite.java>
> and I know the unaligned checking is at
> *common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java*
> <https://github.com/apache/spark/blob/master/common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java>
>
> Either our JDK's method is returning false incorrectly or this test isn't
> using unaligned memory access (so the requirement is invalid), there's no
> mention of alignment in the test itself.
>
> Any guidance would be very much appreciated, cheers
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
Re: BytesToBytes and unaligned memory
Posted by Ted Yu <yu...@gmail.com>.
Can you clarify whether BytesToBytesMapOffHeapSuite passed or failed with
the forced true value for unaligned ?
If the test failed, please pastebin the failure(s).
Thanks
On Fri, Apr 15, 2016 at 8:32 AM, Adam Roberts <AR...@uk.ibm.com> wrote:
> Ted, yep I'm working from the latest code which includes that unaligned
> check, for experimenting I've modified that code to ignore the unaligned
> check (just go ahead and say we support it anyway, even though our JDK
> returns false: the return value of java.nio.Bits.unaligned()).
>
> My Platform.java for testing contains:
>
> private static final boolean unaligned;
>
> static {
> boolean _unaligned;
> // use reflection to access unaligned field
> try {
> * System.out.println("Checking unaligned support");*
> Class<?> bitsClass =
> Class.forName("java.nio.Bits", false,
> ClassLoader.getSystemClassLoader());
> Method unalignedMethod = bitsClass.getDeclaredMethod("unaligned");
> unalignedMethod.setAccessible(true);
> _unaligned = Boolean.TRUE.equals(unalignedMethod.invoke(null));
> *System.out.println("Used reflection and _unaligned is: " +
> _unaligned);*
> * System.out.println("Setting to true anyway for experimenting");*
> * _unaligned = true;*
> } catch (Throwable t) {
> // We at least know x86 and x64 support unaligned access.
> String arch = System.getProperty("os.arch", "");
> //noinspection DynamicRegexReplaceableByCompiledPattern
> * // We don't actually get here since we find the unaligned method OK
> and it returns false (I override with true anyway)*
> * // but add s390x incase we somehow fail anyway.*
> * System.out.println("Checking for s390x, os.arch is: " + arch);*
> * _unaligned =
> arch.matches("^(i[3-6]86|x86(_64)?|x64|s390x|amd64)$");*
> }
> unaligned = _unaligned;
> * System.out.println("returning: " + unaligned);*
> }
> }
>
> Output is, as you'd expect, "used reflection and _unaligned is false,
> setting to true anyway for experimenting", and the tests pass.
>
> No other problems on the platform (pending a different pull request).
>
> Cheers,
>
>
>
>
>
>
>
> From: Ted Yu <yu...@gmail.com>
> To: Adam Roberts/UK/IBM@IBMGB
> Cc: "dev@spark.apache.org" <de...@spark.apache.org>
> Date: 15/04/2016 15:32
> Subject: Re: BytesToBytes and unaligned memory
> ------------------------------
>
>
>
> I assume you tested 2.0 with SPARK-12181 .
>
> Related code from Platform.java if java.nio.Bits#unaligned() throws
> exception:
>
> // We at least know x86 and x64 support unaligned access.
> String arch = System.getProperty("os.arch", "");
> //noinspection DynamicRegexReplaceableByCompiledPattern
> _unaligned = arch.matches("^(i[3-6]86|x86(_64)?|x64|amd64)$");
>
> Can you give us some detail on how the code runs for JDKs on zSystems ?
>
> Thanks
>
> On Fri, Apr 15, 2016 at 7:01 AM, Adam Roberts <*AROBERTS@uk.ibm.com*
> <AR...@uk.ibm.com>> wrote:
> Hi, I'm testing Spark 2.0.0 on various architectures and have a question,
> are we sure if
> *core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java*
> <https://github.com/apache/spark/blob/96941b12f8b465df21423275f3cd3ade579b4fa1/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java>
> really is attempting to use unaligned memory access (for the
> BytesToBytesMapOffHeapSuite tests specifically)?
>
> Our JDKs on zSystems for example return false for the
> java.nio.Bits.unaligned() method and yet if I skip this check and add s390x
> to the supported architectures (for zSystems), all thirteen tests here
> pass.
>
> The 13 tests here all fail as we do not pass the unaligned requirement
> (but perhaps incorrectly):
>
> *core/src/test/java/org/apache/spark/unsafe/map/BytesToBytesMapOffHeapSuite.java*
> <https://github.com/apache/spark/blob/d6dc12ef0146ae409834c78737c116050961f350/core/src/test/java/org/apache/spark/unsafe/map/BytesToBytesMapOffHeapSuite.java>
> and I know the unaligned checking is at
> *common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java*
> <https://github.com/apache/spark/blob/master/common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java>
>
> Either our JDK's method is returning false incorrectly or this test isn't
> using unaligned memory access (so the requirement is invalid), there's no
> mention of alignment in the test itself.
>
> Any guidance would be very much appreciated, cheers
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
Re: BytesToBytes and unaligned memory
Posted by Ted Yu <yu...@gmail.com>.
I assume you tested 2.0 with SPARK-12181 .
Related code from Platform.java if java.nio.Bits#unaligned() throws
exception:
// We at least know x86 and x64 support unaligned access.
String arch = System.getProperty("os.arch", "");
//noinspection DynamicRegexReplaceableByCompiledPattern
_unaligned = arch.matches("^(i[3-6]86|x86(_64)?|x64|amd64)$");
Can you give us some detail on how the code runs for JDKs on zSystems ?
Thanks
On Fri, Apr 15, 2016 at 7:01 AM, Adam Roberts <AR...@uk.ibm.com> wrote:
> Hi, I'm testing Spark 2.0.0 on various architectures and have a question,
> are we sure if
> core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
> <https://github.com/apache/spark/blob/96941b12f8b465df21423275f3cd3ade579b4fa1/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java>
> really is attempting to use unaligned memory access (for the
> BytesToBytesMapOffHeapSuite tests specifically)?
>
> Our JDKs on zSystems for example return false for the
> java.nio.Bits.unaligned() method and yet if I skip this check and add s390x
> to the supported architectures (for zSystems), all thirteen tests here
> pass.
>
> The 13 tests here all fail as we do not pass the unaligned requirement
> (but perhaps incorrectly):
>
> core/src/test/java/org/apache/spark/unsafe/map/BytesToBytesMapOffHeapSuite.java
> <https://github.com/apache/spark/blob/d6dc12ef0146ae409834c78737c116050961f350/core/src/test/java/org/apache/spark/unsafe/map/BytesToBytesMapOffHeapSuite.java>
> and I know the unaligned checking is at
> common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java
> <https://github.com/apache/spark/blob/master/common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java>
>
> Either our JDK's method is returning false incorrectly or this test isn't
> using unaligned memory access (so the requirement is invalid), there's no
> mention of alignment in the test itself.
>
> Any guidance would be very much appreciated, cheers
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
Re: Unable to access Resource Manager /Name Node on port 9026 / 9101
on a Spark EMR Cluster
Posted by Jonathan Kelly <jo...@gmail.com>.
Ever since emr-4.x, the service ports have been synced as much as possible
with open source, so the YARN ResourceManager UI is on port 8088, and the
NameNode UI is on port 50070. See
http://docs.aws.amazon.com/ElasticMapReduce/latest/ReleaseGuide/emr-release-differences.html#d0e23719
for
more information.
~ Jonathan
On Fri, Apr 15, 2016 at 7:29 AM Chadha Pooja <Ch...@bcg.com> wrote:
> Hi ,
>
>
>
>
>
> We have setup a Spark Cluster (3 node) on Amazon EMR.
>
>
>
> We aren’t able to use port 9026 and 9101 on the existing Spark EMR Cluster
> which are part of the Web UIs offered with Amazon EMR. I was able to use
> other ports like Zeppelin port, 8890, HUE etc
>
>
>
> We checked that the security settings currently are open to everyone, and
> it is not an issue with security.
>
>
>
> URLs
>
>
>
> Hadoop ResourceManager
>
> http://master-node-IP:9026/
>
> Hadoop HDFS NameNode
>
> http://master-node-IP:9101/
>
>
>
> Errors Observed on Fiddler:
>
> *Port 9026: *
>
> [Fiddler] The connection to 'masternodeIP' failed.
> Error: TimedOut (0x274c).
> System.Net.Sockets.SocketException A connection attempt failed because the
> connected party did not properly respond after a period of time, or
> established connection failed because connected host has failed to respond
> <<MasternodeIP>>:9026
>
>
>
> *Port 9101:*
>
> [Fiddler] The connection to <<MasternodeIP>>: failed.
> Error: TimedOut (0x274c).
> System.Net.Sockets.SocketException A connection attempt failed because the
> connected party did not properly respond after a period of time, or
> established connection failed because connected host has failed to respond
> <<MasternodeIP>>:9101
>
>
>
> Does anyone have any experiences or pointers? Appreciate your help!
>
>
>
> Thanks!
>
>
>
> ------------------------------
>
> The Boston Consulting Group, Inc.
>
> This e-mail message may contain confidential and/or privileged
> information. If you are not an addressee or otherwise authorized to receive
> this message, you should not use, copy, disclose or take any action based
> on this e-mail or any information contained in the message. If you have
> received this material in error, please advise the sender immediately by
> reply e-mail and delete this message. Thank you.
>
Re: Unable to access Resource Manager /Name Node on port 9026 / 9101
on a Spark EMR Cluster
Posted by Wei-Shun Lo <ra...@gmail.com>.
Hi Chanda,
You may want to check by using nmap to check whether the port and service
is correctly started locally.
ex. nmap localhost
If the port is already successfully internally, it might be related to the
outbound/inbound traffic control in your security group setting.
Just fyi.
On Fri, Apr 15, 2016 at 7:29 AM, Chadha Pooja <Ch...@bcg.com> wrote:
> Hi ,
>
>
>
>
>
> We have setup a Spark Cluster (3 node) on Amazon EMR.
>
>
>
> We aren’t able to use port 9026 and 9101 on the existing Spark EMR Cluster
> which are part of the Web UIs offered with Amazon EMR. I was able to use
> other ports like Zeppelin port, 8890, HUE etc
>
>
>
> We checked that the security settings currently are open to everyone, and
> it is not an issue with security.
>
>
>
> URLs
>
>
>
> Hadoop ResourceManager
>
> http://master-node-IP:9026/
>
> Hadoop HDFS NameNode
>
> http://master-node-IP:9101/
>
>
>
> Errors Observed on Fiddler:
>
> *Port 9026: *
>
> [Fiddler] The connection to 'masternodeIP' failed.
> Error: TimedOut (0x274c).
> System.Net.Sockets.SocketException A connection attempt failed because the
> connected party did not properly respond after a period of time, or
> established connection failed because connected host has failed to respond
> <<MasternodeIP>>:9026
>
>
>
> *Port 9101:*
>
> [Fiddler] The connection to <<MasternodeIP>>: failed.
> Error: TimedOut (0x274c).
> System.Net.Sockets.SocketException A connection attempt failed because the
> connected party did not properly respond after a period of time, or
> established connection failed because connected host has failed to respond
> <<MasternodeIP>>:9101
>
>
>
> Does anyone have any experiences or pointers? Appreciate your help!
>
>
>
> Thanks!
>
>
>
> ------------------------------
>
> The Boston Consulting Group, Inc.
>
> This e-mail message may contain confidential and/or privileged
> information. If you are not an addressee or otherwise authorized to receive
> this message, you should not use, copy, disclose or take any action based
> on this e-mail or any information contained in the message. If you have
> received this material in error, please advise the sender immediately by
> reply e-mail and delete this message. Thank you.
>
--
Best Luck,
Ralic Lo
*------------------------------------------------------------------------*
*---*
Phone: 408-609-7628
Email: RalicLo@gmail.com
*------------------------------------------------------------------------*
*---*
*INSPIRATION FILLS THE GAP OF KNOWLEDGE !*
Unable to access Resource Manager /Name Node on port 9026 / 9101 on a
Spark EMR Cluster
Posted by Chadha Pooja <Ch...@bcg.com>.
Hi ,
We have setup a Spark Cluster (3 node) on Amazon EMR.
We aren't able to use port 9026 and 9101 on the existing Spark EMR Cluster which are part of the Web UIs offered with Amazon EMR. I was able to use other ports like Zeppelin port, 8890, HUE etc
We checked that the security settings currently are open to everyone, and it is not an issue with security.
URLs
Hadoop ResourceManager
http://master-node-IP:9026/
Hadoop HDFS NameNode
http://master-node-IP:9101/
Errors Observed on Fiddler:
Port 9026:
[Fiddler] The connection to 'masternodeIP' failed.
Error: TimedOut (0x274c).
System.Net.Sockets.SocketException A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond <<MasternodeIP>>:9026
Port 9101:
[Fiddler] The connection to <<MasternodeIP>>: failed.
Error: TimedOut (0x274c).
System.Net.Sockets.SocketException A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond <<MasternodeIP>>:9101
Does anyone have any experiences or pointers? Appreciate your help!
Thanks!
______________________________________________________________________________
The Boston Consulting Group, Inc.
This e-mail message may contain confidential and/or privileged information.
If you are not an addressee or otherwise authorized to receive this message,
you should not use, copy, disclose or take any action based on this e-mail or
any information contained in the message. If you have received this material
in error, please advise the sender immediately by reply e-mail and delete this
message. Thank you.