You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by kiszk <gi...@git.apache.org> on 2017/05/20 10:07:51 UTC
[GitHub] spark pull request #18042: [SPARK-20817][core] Benchmark.getProcessorName() ...
GitHub user kiszk opened a pull request:
https://github.com/apache/spark/pull/18042
[SPARK-20817][core] Benchmark.getProcessorName() returns "Unknown processor" on ppc and 390 platforms
## What changes were proposed in this pull request?
This PR fixes that `Benchmark.getProcessorName()` returns `Unknown processor` string on ppc and 390 Linux platforms.
After applying this PR, `Benchmark.getProcessorName()` on two Linux platforms return the following strings:
ppc64le:
```
POWER8E (raw), altivec supported @ 3690.000000MHz
```
s390x:
```
IBM/S390, identification = 01AFF7, machine = 2964
```
## How was this patch tested?
Manually tested on ppc64le and s390x Linux platforms
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kiszk/spark SPARK-20817
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18042.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18042
----
commit 07a71c59ca45fb3c4e37bc6b58c24242f42c7a1a
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2017-05-20T10:02:47Z
initial commit
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18042: [SPARK-20817][core] Fix to return "Unknown processor" on...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18042
Yeah I get that, but does it cause a problem? it won't if it's just for display, but that's what I'm asking.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18042: [SPARK-20817][core] Fix to return "Unknown processor" on...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/18042
@srowen What do you mean "consistent"? Is it good to use a consistent format "processor identification information @ clock"?
This PR make outputs of x86_64 and ppc64le similar. I could output "IBM/S390, identification = 01AFF7, machine = 2964 @ unknown clock" for s390x.
Any other great ideas are appreciated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18042: [SPARK-20817][core] Fix to return "Unknown processor" on...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/18042
It is due to differences among CPU architectures. Here are original outputs of `/proc/info`
It is used to identify CPU architecture and its model.
**POWER**
```
> cat /proc/cpuinfo
processor : 0
cpu : POWER8E (raw), altivec supported
clock : 3690.000000MHz
revision : 2.1 (pvr 004b 0201)
processor : 1
cpu : POWER8E (raw), altivec supported
clock : 3690.000000MHz
revision : 2.1 (pvr 004b 0201)
processor : 2
cpu : POWER8E (raw), altivec supported
clock : 3690.000000MHz
revision : 2.1 (pvr 004b 0201)
...
processor : 158
cpu : POWER8E (raw), altivec supported
clock : 3690.000000MHz
revision : 2.1 (pvr 004b 0201)
processor : 159
cpu : POWER8E (raw), altivec supported
clock : 3690.000000MHz
revision : 2.1 (pvr 004b 0201)
timebase : 512000000
platform : PowerNV
model : 8247-42L
machine : PowerNV 8247-42L
firmware : OPAL v3
```
**S390**
```
> cat /proc/cpuinfo
vendor_id : IBM/S390
# processors : 32
bogomips per cpu: 20325.00
features : esan3 zarch stfle msa ldisp eimm dfp edat etf3eh highgprs te vx
cache0 : level=1 type=Data scope=Private size=128K line_size=256 associativity=8
cache1 : level=1 type=Instruction scope=Private size=96K line_size=256 associativity=6
cache2 : level=2 type=Data scope=Private size=2048K line_size=256 associativity=8
cache3 : level=2 type=Instruction scope=Private size=2048K line_size=256 associativity=8
cache4 : level=3 type=Unified scope=Shared size=65536K line_size=256 associativity=16
cache5 : level=4 type=Unified scope=Shared size=491520K line_size=256 associativity=30
processor 0: version = 00, identification = 01AFF7, machine = 2964
processor 1: version = 00, identification = 01AFF7, machine = 2964
processor 2: version = 00, identification = 01AFF7, machine = 2964
processor 3: version = 00, identification = 01AFF7, machine = 2964
processor 4: version = 00, identification = 01AFF7, machine = 2964
processor 5: version = 00, identification = 01AFF7, machine = 2964
processor 6: version = 00, identification = 01AFF7, machine = 2964
processor 7: version = 00, identification = 01AFF7, machine = 2964
processor 8: version = 00, identification = 01AFF7, machine = 2964
processor 9: version = 00, identification = 01AFF7, machine = 2964
processor 10: version = 00, identification = 01AFF7, machine = 2964
processor 11: version = 00, identification = 01AFF7, machine = 2964
processor 12: version = 00, identification = 01AFF7, machine = 2964
processor 13: version = 00, identification = 01AFF7, machine = 2964
processor 14: version = 00, identification = 01AFF7, machine = 2964
processor 15: version = 00, identification = 01AFF7, machine = 2964
processor 16: version = 00, identification = 01AFF7, machine = 2964
processor 17: version = 00, identification = 01AFF7, machine = 2964
processor 18: version = 00, identification = 01AFF7, machine = 2964
processor 19: version = 00, identification = 01AFF7, machine = 2964
processor 20: version = 00, identification = 01AFF7, machine = 2964
processor 21: version = 00, identification = 01AFF7, machine = 2964
processor 22: version = 00, identification = 01AFF7, machine = 2964
processor 23: version = 00, identification = 01AFF7, machine = 2964
processor 24: version = 00, identification = 01AFF7, machine = 2964
processor 25: version = 00, identification = 01AFF7, machine = 2964
processor 26: version = 00, identification = 01AFF7, machine = 2964
processor 27: version = 00, identification = 01AFF7, machine = 2964
processor 28: version = 00, identification = 01AFF7, machine = 2964
processor 29: version = 00, identification = 01AFF7, machine = 2964
processor 30: version = 00, identification = 01AFF7, machine = 2964
processor 31: version = 00, identification = 01AFF7, machine = 2964
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18042: [SPARK-20817][core] Fix to return "Unknown processor" on...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18042
@kiszk unless you want to make the output of this function consistent, I'd say I'm not clear that this is a good change
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18042: [SPARK-20817][core] Benchmark.getProcessorName() returns...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18042
**[Test build #77119 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77119/testReport)** for PR 18042 at commit [`07a71c5`](https://github.com/apache/spark/commit/07a71c59ca45fb3c4e37bc6b58c24242f42c7a1a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18042: [SPARK-20817][core] Fix to return "Unknown proces...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk closed the pull request at:
https://github.com/apache/spark/pull/18042
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18042: [SPARK-20817][core] Benchmark.getProcessorName() returns...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18042
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77119/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18042: [SPARK-20817][core] Fix to return "Unknown processor" on...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/18042
I see. I will try to use "cpu @ clock" format.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18042: [SPARK-20817][core] Fix to return "Unknown processor" on...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18042
Does it matter that the format of the output isn't quite in the same format - is it just used for display?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18042: [SPARK-20817][core] Fix to return "Unknown processor" on...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/18042
@srowen @rxin Do we still want this？
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18042: [SPARK-20817][core] Fix to return "Unknown processor" on...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/18042
ping @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18042: [SPARK-20817][core] Benchmark.getProcessorName() returns...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18042
**[Test build #77119 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77119/testReport)** for PR 18042 at commit [`07a71c5`](https://github.com/apache/spark/commit/07a71c59ca45fb3c4e37bc6b58c24242f42c7a1a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18042: [SPARK-20817][core] Fix to return "Unknown processor" on...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/18042
Does this really matter? I'd rather not complicate the actual code for it to display properly in some niche hardware that very few people use.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18042: [SPARK-20817][core] Fix to return "Unknown processor" on...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/18042
Please let's not waste more time here. I don't think the gain is worth the effort required (or even the discussions here).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18042: [SPARK-20817][core] Fix to return "Unknown processor" on...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18042
For example, S390 appears to output "cpu, machine", not "cpu @ clock". If this is actually used by code somewhere it needs to be consistent right? if it's just for display, it's a nice-to-have, but still, seems nicer to output similar strings?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18042: [SPARK-20817][core] Benchmark.getProcessorName() returns...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18042
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org