You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/08/06 06:17:08 UTC
[GitHub] [incubator-pinot] adriancole opened a new pull request #5818: Makes Pinot work on Alpine Linux or Distroless + BusyBox
adriancole opened a new pull request #5818:
URL: https://github.com/apache/incubator-pinot/pull/5818
This fixes the shell template so that it uses /bin/sh not bash. In doing
so, this allows a couple hundred meg smaller image, specifically Alpine
Linus.
Fixes #5809
## Description
Add a description of your PR here.
A good description should include pointers to an issue or design document, etc.
## Upgrade Notes
Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)
* [ ] Yes (Please label as **<code>backward-incompat</code>**, and complete the section below on Release Notes)
Does this PR fix a zero-downtime upgrade introduced earlier?
* [ ] Yes (Please label this as **<code>backward-incompat</code>**, and complete the section below on Release Notes)
Does this PR otherwise need attention when creating release notes? Things to consider:
- New configuration options
- Deprecation of configurations
- Signature changes to public methods/interfaces
- New plugins added or old plugins removed
* [ ] Yes (Please label this PR as **<code>release-notes</code>** and complete the section on Release Notes)
## Release Notes
If you have tagged this as either backward-incompat or release-notes,
you MUST add text here that you would like to see appear in release notes of the
next release.
If you have a series of commits adding or enabling a feature, then
add this section only in final commit that marks the feature completed.
Refer to earlier release notes to see examples of text
## Documentation
If you have introduced a new feature or configuration, please add it to the documentation as well.
See https://docs.pinot.apache.org/developers/developers-and-contributors/update-document
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on pull request #5818: Makes Pinot work on Alpine Linux or Distroless + BusyBox
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on pull request #5818:
URL: https://github.com/apache/incubator-pinot/pull/5818#issuecomment-672728402
> PS I had a bug in my comment [#5818 (comment)](https://github.com/apache/incubator-pinot/pull/5818#issuecomment-670817753) now is fixed. I didn't notice the fork about JRE 9+ needing plugins also in classpath. since this script is independent of this change more an FYI
Got it. I haven't made pinot build succeed from jdk 9.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] adriancole commented on pull request #5818: Makes Pinot work on Alpine Linux or Distroless + BusyBox
Posted by GitBox <gi...@apache.org>.
adriancole commented on pull request #5818:
URL: https://github.com/apache/incubator-pinot/pull/5818#issuecomment-670818315
for completion, here's a Dockerfile that uses it. Maybe not perfect, but possibly helpful in showing where the script is useful.
```Dockerfile
FROM alpine:3.12 as pinot
ARG USER=apache
ARG TAG=master-SNAPSHOT
WORKDIR /pinot
# Jitpack makes it possible to grab a file built on-demand from a branch or a commit
# Effectively it caches the result of maven, so that your laptop doesn't burn up so much building!
# Ex. https://jitpack.io/#apache/incubator-pinot
RUN jitpack() { lib=$1; dir=$2; mkdir -p $dir; wget -q -P $dir https://jitpack.io/com/github/$USER/incubator-pinot/$lib/$TAG/$lib-$TAG-shaded.jar ; } ; \
# Pinot requires the main jar
jitpack pinot-distribution lib && \
# Add plugins you want
jitpack pinot-kafka-2.0 plugins/pinot-stream-ingestion/pinot-kafka-2.0
FROM openzipkin/jre-full:14.0.2-14.29.23
ENV PINOT_HOME=/pinot
ENV PINOT_VAR=/var/pinot
ENV JAVA_OPTS="-Xms128M -Xmx192M -XX:MaxDirectMemorySize=64M -XX:+ExitOnOutOfMemoryError"
WORKDIR ${PINOT_HOME}
# Ensure pinot doesn't run as root
RUN adduser -g '' -h ${PINOT_HOME} -D pinot && \
mkdir -p ${PINOT_VAR}/server/data ${PINOT_VAR}/controller/data && \
chown -R pinot ${PINOT_VAR}
USER pinot
COPY --from=pinot --chown=pinot /pinot ${PINOT_HOME}
# Add patched admin script so that it works with /bin/sh
COPY --chown=pinot pinot-admin.sh ${PINOT_HOME}/bin/pinot-admin.sh
# expose ports for controller/broker/server/admin
EXPOSE 9000 8099 8098 8097 8096 9514
ENTRYPOINT ["./bin/pinot-admin.sh"]
CMD ["run"]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 merged pull request #5818: Makes Pinot work on Alpine Linux or Distroless + BusyBox
Posted by GitBox <gi...@apache.org>.
fx19880617 merged pull request #5818:
URL: https://github.com/apache/incubator-pinot/pull/5818
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] adriancole commented on pull request #5818: Makes Pinot work on Alpine Linux or Distroless + BusyBox
Posted by GitBox <gi...@apache.org>.
adriancole commented on pull request #5818:
URL: https://github.com/apache/incubator-pinot/pull/5818#issuecomment-670817753
fyi I'm currently inlining a script as when you consider docker knows the only image it will run on, the script doesn't need to do work like detect OS, JVM etc.
In this case, I'm ok closing this out if it isn't working.
```sh
#!/bin/sh
# TODO: link to pinot issue why we need internal access
exec java \
--add-opens java.base/jdk.internal.ref=ALL-UNNAMED \
-classpath 'lib/*' \
$JAVA_OPTS \
-Dapp.name=pinot-admin \
-Dapp.pid=$$ \
-Dapp.repo=$PINOT_HOME/lib \
-Dapp.home=$PINOT_HOME \
-Dbasedir=$PINOT_HOME \
-Dplugins.dir=$PINOT_HOME/plugins \
org.apache.pinot.tools.admin.PinotAdministrator \
"$@"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on pull request #5818: Makes Pinot work on Alpine Linux or Distroless + BusyBox
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on pull request #5818:
URL: https://github.com/apache/incubator-pinot/pull/5818#issuecomment-670823578
Got it.
In this case, probably we can put the code here: `https://github.com/apache/incubator-pinot/tree/master/docker/images/pinot`
Maybe create a directory `alpine` and put `Dockerfile` as well as `.sh` file into that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] adriancole commented on pull request #5818: Makes Pinot work on Alpine Linux or Distroless + BusyBox
Posted by GitBox <gi...@apache.org>.
adriancole commented on pull request #5818:
URL: https://github.com/apache/incubator-pinot/pull/5818#issuecomment-672845582
thanks for the fast turnaround and the help fixing the branch to make it work @fx19880617
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 edited a comment on pull request #5818: Makes Pinot work on Alpine Linux or Distroless + BusyBox
Posted by GitBox <gi...@apache.org>.
fx19880617 edited a comment on pull request #5818:
URL: https://github.com/apache/incubator-pinot/pull/5818#issuecomment-672728402
> PS I had a bug in my comment [#5818 (comment)](https://github.com/apache/incubator-pinot/pull/5818#issuecomment-670817753) now is fixed. I didn't notice the fork about JRE 9+ needing plugins also in classpath. since this script is independent of this change more an FYI
Got it. I haven't made pinot build succeed from jdk 9, but runtime should be fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] adriancole commented on pull request #5818: Makes Pinot work on Alpine Linux or Distroless + BusyBox
Posted by GitBox <gi...@apache.org>.
adriancole commented on pull request #5818:
URL: https://github.com/apache/incubator-pinot/pull/5818#issuecomment-672397474
PS I had a bug in my comment https://github.com/apache/incubator-pinot/pull/5818#issuecomment-670817753 now is fixed. I didn't notice the fork about JRE 9+ needing plugins also in classpath. since this script is independent of this change more an FYI
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] adriancole commented on pull request #5818: Makes Pinot work on Alpine Linux or Distroless + BusyBox
Posted by GitBox <gi...@apache.org>.
adriancole commented on pull request #5818:
URL: https://github.com/apache/incubator-pinot/pull/5818#issuecomment-671135796
Tested on Zulu+Alpine JRE 1.8 and Distroless+BusyBox JRE 14 :shipit:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on pull request #5818: Makes Pinot work on Alpine Linux or Distroless + BusyBox
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on pull request #5818:
URL: https://github.com/apache/incubator-pinot/pull/5818#issuecomment-670853909
Updated this PR to make the .sh works.
Could you also verify if your image could reuse this pinot-admin.sh?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] adriancole edited a comment on pull request #5818: Makes Pinot work on Alpine Linux or Distroless + BusyBox
Posted by GitBox <gi...@apache.org>.
adriancole edited a comment on pull request #5818:
URL: https://github.com/apache/incubator-pinot/pull/5818#issuecomment-670817753
fyi I'm currently inlining a script as when you consider docker knows the only image it will run on, the script doesn't need to do work like detect OS, JVM etc.
In this case, I'm ok closing this out if it isn't working.
```sh
#!/bin/sh
# For java 9 and later version, we need to explicitly set Pinot Plugins directory into classpath.
CLASSPATH=$(find lib plugins -type f|xargs echo|tr ' ' ':')
# TODO: link to pinot issue why we need internal access
exec java \
--add-opens java.base/jdk.internal.ref=ALL-UNNAMED \
-classpath $CLASSPATH \
$JAVA_OPTS \
-Dlog4j.configurationFile=etc/log4j2.properties \
-Dapp.name=pinot-admin \
-Dapp.pid=$$ \
-Dapp.repo=$PINOT_HOME/lib \
-Dapp.home=$PINOT_HOME \
-Dbasedir=$PINOT_HOME \
org.apache.pinot.tools.admin.PinotAdministrator \
"$@"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on pull request #5818: Makes Pinot work on Alpine Linux or Distroless + BusyBox
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on pull request #5818:
URL: https://github.com/apache/incubator-pinot/pull/5818#issuecomment-670664334
I'm seeing this error in quickstart ci:
```
***** Launch data ingestion job to build index segment for baseballStats and push to controller *****
##[error]Exception in thread "main" java.lang.RuntimeException: Failed to create IngestionJobRunner instance for class - org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner
at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:130)
at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.runIngestionJob(IngestionJobLauncher.java:110)
at org.apache.pinot.tools.admin.command.QuickstartRunner.launchDataIngestionJob(QuickstartRunner.java:199)
at org.apache.pinot.tools.Quickstart.execute(Quickstart.java:180)
at org.apache.pinot.tools.Quickstart.main(Quickstart.java:222)
Caused by: java.lang.ClassNotFoundException: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at org.apache.pinot.spi.plugin.PluginClassLoader.loadClass(PluginClassLoader.java:80)
at org.apache.pinot.spi.plugin.PluginManager.createInstance(PluginManager.java:268)
at org.apache.pinot.spi.plugin.PluginManager.createInstance(PluginManager.java:239)
at org.apache.pinot.spi.plugin.PluginManager.createInstance(PluginManager.java:220)
at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:128)
... 4 more
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org