You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by GitBox <gi...@apache.org> on 2022/05/23 18:29:03 UTC
[GitHub] [parquet-mr] steveloughran opened a new pull request, #970: PARQUET-2150: parquet-protobuf to compile on Mac M1
steveloughran opened a new pull request, #970:
URL: https://github.com/apache/parquet-mr/pull/970
Force use of x86 protocol executable in builds onl macOS systems,
including those with ARM64 architectures, for which native binaries
do not exist.
### Jira
- [X] My PR addresses the following [Parquet Jira](https://issues.apache.org/jira/browse/PARQUET/) issues and references them in the PR title. For example, "PARQUET-1234: My Parquet PR"
- https://issues.apache.org/jira/browse/PARQUET-XXX
- In case you are adding a dependency, check if the license complies with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
### Tests
- [X] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
It can only be validated on an M1 mac. the normal CI test runs will verify the changes do not cause regressions elsewhere.
Note: this patch was based on hadoop's HADOOP-17939 patch https://github.com/apache/hadoop/pull/3486 -this was needed to fix the hadoop build
### Commits
- [X] My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [X] In case of new functionality, my PR adds documentation that describes how to use it.
- All the public functions and the classes in the PR contain Javadoc that explain what it does
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] sunchao commented on pull request #970: PARQUET-2150: parquet-protobuf to compile on Mac M1
Posted by GitBox <gi...@apache.org>.
sunchao commented on PR #970:
URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1159806290
@shangxinli my approach is different from @steveloughran 's one. Since newer version of protobuf already provides M1 artifacts, upgrade will solve the issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] steveloughran commented on pull request #970: PARQUET-2150: parquet-protobuf to compile on Mac M1
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #970:
URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1136296983
its the maven protoc compiler plugin; wants to pull in something native. i have no idea why, but do know, if it is not there, you get to see a stack trace.
```
[ERROR] Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run (generate-sources) on project parquet-protobuf: Error resolving artifact: com.google.protobuf:protoc:3.16.1: Could not find artifact com.google.protobuf:protoc:exe:osx-aarch_64:3.16.1 in jitpack.io (https://jitpack.io)
[ERROR]
[ERROR] Try downloading the file manually from the project website.
[ERROR]
[ERROR] Then, install it using the command:
[ERROR] mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.16.1 -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=/path/to/file
[ERROR]
[ERROR] Alternatively, if you host your own repository you can deploy the file there:
[ERROR] mvn deploy:deploy-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.16.1 -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[ERROR]
[ERROR]
[ERROR] com.google.protobuf:protoc:exe:3.16.1
[ERROR]
[ERROR] from the specified remote repositories:
[ERROR] jitpack.io (https://jitpack.io, releases=true, snapshots=true),
[ERROR] apache.snapshots (https://repository.apache.org/snapshots, releases=false, snapshots=true),
[ERROR] maven-central (https://repo1.maven.org/maven2, releases=true, snapshots=false)
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <args> -rf :parquet-protobuf
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] steveloughran commented on pull request #970: PARQUET-2150: parquet-protobuf to compile on Mac M1
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #970:
URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1135004012
fyi @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] steveloughran commented on pull request #970: PARQUET-2150: parquet-protobuf to compile on Mac M1
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #970:
URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1189454282
resolved by #973
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] theosib-amazon commented on pull request #970: PARQUET-2150: parquet-protobuf to compile on Mac M1
Posted by GitBox <gi...@apache.org>.
theosib-amazon commented on PR #970:
URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1136102990
I looked up parquet-protobuf and only found Java code. Is there some use of JNI here that I'm not finding?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] steveloughran commented on pull request #970: PARQUET-2150: parquet-protobuf to compile on Mac M1
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #970:
URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1160638077
this patch is based on Dongjoon;s one for hadoop, tells maven to use the x86 artifact on macbook m1 builds.
the sunchao one switches to a version of protobuf with a genuine mac m1 artifacts, a version which should also include some CVE fixes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] sunchao commented on pull request #970: PARQUET-2150: parquet-protobuf to compile on Mac M1
Posted by GitBox <gi...@apache.org>.
sunchao commented on PR #970:
URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1151738855
Oops @steveloughran just found out this PR. I opened #973 for the same purpose
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] shangxinli commented on pull request #970: PARQUET-2150: parquet-protobuf to compile on Mac M1
Posted by GitBox <gi...@apache.org>.
shangxinli commented on PR #970:
URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1159351267
@sunchao I see your change is to upgrade the protobuf version. Is that required to solve this problem, which I don't see in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] steveloughran commented on pull request #970: PARQUET-2150: parquet-protobuf to compile on Mac M1
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #970:
URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1141216939
ok, somehow I've broken the default classifier logic, which is of course critical.
downgrading this to a draft until it's ready to be reviewed
```
Error: Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run (generate-sources) on project parquet-protobuf: Error resolving artifact: com.google.protobuf:protoc:3.16.1:exe:${os.detected.classifier}: Could not find artifact com.google.protobuf:protoc:exe:${os.detected.classifier}:3.16.1 in jitpack.io (https://jitpack.io/)
Error:
Error: Try downloading the file manually from the project website.
Error:
Error: Then, install it using the command:
Error: mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.16.1 -Dclassifier=${os.detected.classifier} -Dpackaging=exe -Dfile=/path/to/file
Error:
Error: Alternatively, if you host your own repository you can deploy the file there:
Error: mvn deploy:deploy-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.16.1 -Dclassifier=${os.detected.classifier} -Dpackaging=exe -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] steveloughran commented on pull request #970: PARQUET-2150: parquet-protobuf to compile on Mac M1
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #970:
URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1145835507
tested lasted patch on
* macbook m1
* macbook x86
* ec2 x86
i would set up my raspberry pi for a build too -I just fear having to build the thrift compiler
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] steveloughran closed pull request #970: PARQUET-2150: parquet-protobuf to compile on Mac M1
Posted by GitBox <gi...@apache.org>.
steveloughran closed pull request #970: PARQUET-2150: parquet-protobuf to compile on Mac M1
URL: https://github.com/apache/parquet-mr/pull/970
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] shangxinli commented on pull request #970: PARQUET-2150: parquet-protobuf to compile on Mac M1
Posted by GitBox <gi...@apache.org>.
shangxinli commented on PR #970:
URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1172920665
@steveloughran Thanks for the explanation! Do you have concerns if we use [PR-973](https://github.com/apache/parquet-mr/pull/973)? It seems we can rely on proto-buf itself to solve the issue.
@sunchao please add @steveloughran as co-author for your PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] steveloughran commented on pull request #970: PARQUET-2150: parquet-protobuf to compile on Mac M1
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #970:
URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1177929495
oh, upgrading is better!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] dongjoon-hyun commented on pull request #970: PARQUET-2150: parquet-protobuf to compile on Mac M1
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #970:
URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1135005341
Thank you for pinging me, @steveloughran .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org