You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by GitBox <gi...@apache.org> on 2022/02/02 09:49:25 UTC
[GitHub] [drill] jnturton opened a new pull request #2447: DRILL-7994: Dependency version updates for severe vulnerabilities
jnturton opened a new pull request #2447:
URL: https://github.com/apache/drill/pull/2447
# [DRILL-7994](https://issues.apache.org/jira/browse/DRILL-7994): Dependency version updates for severe vulnerabilities
## Description
Based on the OWASP dependency check report and dependabot.
## Documentation
N/A
## Testing
Existing tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] vdiravka commented on a change in pull request #2447: DRILL-7994: Dependency version updates for severe vulnerabilities
Posted by GitBox <gi...@apache.org>.
vdiravka commented on a change in pull request #2447:
URL: https://github.com/apache/drill/pull/2447#discussion_r797605652
##########
File path: .travis.yml
##########
@@ -50,41 +50,43 @@ before_install:
- git fetch --unshallow
# Install libraries required for protobuf generation
- |
- if [ $PHASE = "build_checkstyle_protobuf" ]; then \
- sudo apt-get install -y libboost-all-dev libzookeeper-mt-dev libsasl2-dev cmake libcppunit-dev checkinstall && \
- pushd .. && \
- if [ -f $HOME/protobuf/protobuf_3.16.1* ]; then \
- sudo dpkg -i $HOME/protobuf/protobuf_3.16.1*; \
- else \
- wget https://github.com/protocolbuffers/protobuf/releases/download/v3.16.1/protobuf-java-3.16.1.zip && \
- unzip protobuf-java-3.16.1.zip && pushd protobuf-3.16.1 && \
- ./configure && sudo make && sudo checkinstall -y && \
- if [ ! -d $HOME/protobuf ]; then \
- mkdir -p $HOME/protobuf; \
- fi && \
- mv protobuf_3.16.1* $HOME/protobuf/ && popd; \
- fi && \
- sudo ldconfig && popd; \
+ if [ $PHASE = "build_checkstyle_protobuf" ]; then
+ sudo apt-get install -y libboost-all-dev libzookeeper-mt-dev libsasl2-dev cmake libcppunit-dev checkinstall &&
+ pushd .. &&
+ if [ -f $HOME/protobuf/protobuf_3.16.1* ]; then
+ sudo dpkg -i $HOME/protobuf/protobuf_3.16.1*;
+ else
+ wget https://github.com/protocolbuffers/protobuf/releases/download/v3.16.1/protobuf-java-3.16.1.zip &&
+ unzip protobuf-java-3.16.1.zip && pushd protobuf-3.16.1 &&
+ ./configure && sudo make && sudo checkinstall -y &&
+ if [ ! -d $HOME/protobuf ]; then
+ mkdir -p $HOME/protobuf;
+ fi &&
+ mv protobuf_3.16.1* $HOME/protobuf/ && popd;
+ fi &&
+ sudo ldconfig && popd;
fi
install:
# For tests phase runs unit tests
# For protobuf phase: builds Drill project, performs license checkstyle goal and regenerates Java and C++ Protobuf files
- |
- if [ $PHASE = "tests" ]; then \
- mvn install --batch-mode --no-transfer-progress \
- -DexcludedGroups="org.apache.drill.categories.SlowTest,org.apache.drill.categories.UnlikelyTest,org.apache.drill.categories.SecurityTest"; \
- elif [ $PHASE = "build_checkstyle_protobuf" ]; then \
- MAVEN_OPTS="-Xms1G -Xmx1G" mvn install --no-transfer-progress -Drat.skip=false -Dlicense.skip=false --batch-mode -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -DskipTests=true -Dmaven.javadoc.skip=true -Dmaven.source.skip=true && \
- pushd protocol && mvn process-sources -P proto-compile && popd && \
- mkdir contrib/native/client/build && pushd contrib/native/client/build && /usr/bin/cmake -G "Unix Makefiles" .. && make cpProtobufs && popd; \
+ if [ $PHASE = "tests" ]; then
+ echo Selecting parallel GC to minimise peak mem usage.
+ MAVEN_OPTS="-XX:+UseParallelGC"
Review comment:
@jnturton Could you create a task to tune the application to use less memory and back to G1 as default for GitHub CI?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] vdiravka commented on a change in pull request #2447: DRILL-7994: Dependency version updates for severe vulnerabilities
Posted by GitBox <gi...@apache.org>.
vdiravka commented on a change in pull request #2447:
URL: https://github.com/apache/drill/pull/2447#discussion_r797537220
##########
File path: contrib/storage-phoenix/pom.xml
##########
@@ -33,7 +33,7 @@
<phoenix.version>5.1.2</phoenix.version>
<!-- Keep the 2.4.2 to reduce dependency conflict -->
<hbase.minicluster.version>2.4.2</hbase.minicluster.version>
- <jetty.test.version>9.4.31.v20200723</jetty.test.version>
+ <jetty.test.version>9.4.44.v20210927</jetty.test.version>
Review comment:
I've checked and `SecuredPhoenixTestSuite` works fine with this jetty version (dependencies in `drill-storage-phoenix` are still needed for that).
So please remove `jetty.test.version` property and it's usage
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] jnturton commented on pull request #2447: DRILL-7994: Dependency version updates for severe vulnerabilities
Posted by GitBox <gi...@apache.org>.
jnturton commented on pull request #2447:
URL: https://github.com/apache/drill/pull/2447#issuecomment-1027942224
@vvysotskyi sorry, I wrote an @vdiravka but it was you who commented. The idea behind this GC is to reduce memory usage in low mem environments like our CI. The trade off is that it has some worse pause times than the newer GCs but some longer pauses are better than a crashed CI. My prediction is that reverting it will bring back "JVM process terminated unexpectedly" for the JDK 8 build specifically. But it can be set in another PR, I just needed a CI pass here for deps updates reasons. It was just a means to an end for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] jnturton commented on a change in pull request #2447: DRILL-7994: Dependency version updates for severe vulnerabilities
Posted by GitBox <gi...@apache.org>.
jnturton commented on a change in pull request #2447:
URL: https://github.com/apache/drill/pull/2447#discussion_r797589783
##########
File path: .travis.yml
##########
@@ -50,41 +50,43 @@ before_install:
- git fetch --unshallow
# Install libraries required for protobuf generation
- |
- if [ $PHASE = "build_checkstyle_protobuf" ]; then \
- sudo apt-get install -y libboost-all-dev libzookeeper-mt-dev libsasl2-dev cmake libcppunit-dev checkinstall && \
- pushd .. && \
- if [ -f $HOME/protobuf/protobuf_3.16.1* ]; then \
- sudo dpkg -i $HOME/protobuf/protobuf_3.16.1*; \
- else \
- wget https://github.com/protocolbuffers/protobuf/releases/download/v3.16.1/protobuf-java-3.16.1.zip && \
- unzip protobuf-java-3.16.1.zip && pushd protobuf-3.16.1 && \
- ./configure && sudo make && sudo checkinstall -y && \
- if [ ! -d $HOME/protobuf ]; then \
- mkdir -p $HOME/protobuf; \
- fi && \
- mv protobuf_3.16.1* $HOME/protobuf/ && popd; \
- fi && \
- sudo ldconfig && popd; \
+ if [ $PHASE = "build_checkstyle_protobuf" ]; then
+ sudo apt-get install -y libboost-all-dev libzookeeper-mt-dev libsasl2-dev cmake libcppunit-dev checkinstall &&
+ pushd .. &&
+ if [ -f $HOME/protobuf/protobuf_3.16.1* ]; then
+ sudo dpkg -i $HOME/protobuf/protobuf_3.16.1*;
+ else
+ wget https://github.com/protocolbuffers/protobuf/releases/download/v3.16.1/protobuf-java-3.16.1.zip &&
+ unzip protobuf-java-3.16.1.zip && pushd protobuf-3.16.1 &&
+ ./configure && sudo make && sudo checkinstall -y &&
+ if [ ! -d $HOME/protobuf ]; then
+ mkdir -p $HOME/protobuf;
+ fi &&
+ mv protobuf_3.16.1* $HOME/protobuf/ && popd;
+ fi &&
+ sudo ldconfig && popd;
fi
install:
# For tests phase runs unit tests
# For protobuf phase: builds Drill project, performs license checkstyle goal and regenerates Java and C++ Protobuf files
- |
- if [ $PHASE = "tests" ]; then \
- mvn install --batch-mode --no-transfer-progress \
- -DexcludedGroups="org.apache.drill.categories.SlowTest,org.apache.drill.categories.UnlikelyTest,org.apache.drill.categories.SecurityTest"; \
- elif [ $PHASE = "build_checkstyle_protobuf" ]; then \
- MAVEN_OPTS="-Xms1G -Xmx1G" mvn install --no-transfer-progress -Drat.skip=false -Dlicense.skip=false --batch-mode -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -DskipTests=true -Dmaven.javadoc.skip=true -Dmaven.source.skip=true && \
- pushd protocol && mvn process-sources -P proto-compile && popd && \
- mkdir contrib/native/client/build && pushd contrib/native/client/build && /usr/bin/cmake -G "Unix Makefiles" .. && make cpProtobufs && popd; \
+ if [ $PHASE = "tests" ]; then
+ echo Selecting parallel GC to minimise peak mem usage.
+ MAVEN_OPTS="-XX:+UseParallelGC"
Review comment:
@vdiravka it was selected for an extremely memory constrained environment and regular Drill users with such memory constraints are well-advised to make the same selection. Will revert for now...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] jnturton merged pull request #2447: DRILL-7994: Dependency version updates for severe vulnerabilities
Posted by GitBox <gi...@apache.org>.
jnturton merged pull request #2447:
URL: https://github.com/apache/drill/pull/2447
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] vvysotskyi commented on a change in pull request #2447: DRILL-7994: Dependency version updates for severe vulnerabilities
Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2447:
URL: https://github.com/apache/drill/pull/2447#discussion_r797570750
##########
File path: .travis.yml
##########
@@ -50,41 +50,43 @@ before_install:
- git fetch --unshallow
# Install libraries required for protobuf generation
- |
- if [ $PHASE = "build_checkstyle_protobuf" ]; then \
- sudo apt-get install -y libboost-all-dev libzookeeper-mt-dev libsasl2-dev cmake libcppunit-dev checkinstall && \
- pushd .. && \
- if [ -f $HOME/protobuf/protobuf_3.16.1* ]; then \
- sudo dpkg -i $HOME/protobuf/protobuf_3.16.1*; \
- else \
- wget https://github.com/protocolbuffers/protobuf/releases/download/v3.16.1/protobuf-java-3.16.1.zip && \
- unzip protobuf-java-3.16.1.zip && pushd protobuf-3.16.1 && \
- ./configure && sudo make && sudo checkinstall -y && \
- if [ ! -d $HOME/protobuf ]; then \
- mkdir -p $HOME/protobuf; \
- fi && \
- mv protobuf_3.16.1* $HOME/protobuf/ && popd; \
- fi && \
- sudo ldconfig && popd; \
+ if [ $PHASE = "build_checkstyle_protobuf" ]; then
+ sudo apt-get install -y libboost-all-dev libzookeeper-mt-dev libsasl2-dev cmake libcppunit-dev checkinstall &&
+ pushd .. &&
+ if [ -f $HOME/protobuf/protobuf_3.16.1* ]; then
+ sudo dpkg -i $HOME/protobuf/protobuf_3.16.1*;
+ else
+ wget https://github.com/protocolbuffers/protobuf/releases/download/v3.16.1/protobuf-java-3.16.1.zip &&
+ unzip protobuf-java-3.16.1.zip && pushd protobuf-3.16.1 &&
+ ./configure && sudo make && sudo checkinstall -y &&
+ if [ ! -d $HOME/protobuf ]; then
+ mkdir -p $HOME/protobuf;
+ fi &&
+ mv protobuf_3.16.1* $HOME/protobuf/ && popd;
+ fi &&
+ sudo ldconfig && popd;
fi
install:
# For tests phase runs unit tests
# For protobuf phase: builds Drill project, performs license checkstyle goal and regenerates Java and C++ Protobuf files
- |
- if [ $PHASE = "tests" ]; then \
- mvn install --batch-mode --no-transfer-progress \
- -DexcludedGroups="org.apache.drill.categories.SlowTest,org.apache.drill.categories.UnlikelyTest,org.apache.drill.categories.SecurityTest"; \
- elif [ $PHASE = "build_checkstyle_protobuf" ]; then \
- MAVEN_OPTS="-Xms1G -Xmx1G" mvn install --no-transfer-progress -Drat.skip=false -Dlicense.skip=false --batch-mode -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -DskipTests=true -Dmaven.javadoc.skip=true -Dmaven.source.skip=true && \
- pushd protocol && mvn process-sources -P proto-compile && popd && \
- mkdir contrib/native/client/build && pushd contrib/native/client/build && /usr/bin/cmake -G "Unix Makefiles" .. && make cpProtobufs && popd; \
+ if [ $PHASE = "tests" ]; then
+ echo Selecting parallel GC to minimise peak mem usage.
Review comment:
Please avoid adding such comments in the ci output, they might confuse people that trying to figure out why their tests failing.
##########
File path: .travis.yml
##########
@@ -50,41 +50,43 @@ before_install:
- git fetch --unshallow
# Install libraries required for protobuf generation
- |
- if [ $PHASE = "build_checkstyle_protobuf" ]; then \
- sudo apt-get install -y libboost-all-dev libzookeeper-mt-dev libsasl2-dev cmake libcppunit-dev checkinstall && \
- pushd .. && \
- if [ -f $HOME/protobuf/protobuf_3.16.1* ]; then \
- sudo dpkg -i $HOME/protobuf/protobuf_3.16.1*; \
- else \
- wget https://github.com/protocolbuffers/protobuf/releases/download/v3.16.1/protobuf-java-3.16.1.zip && \
- unzip protobuf-java-3.16.1.zip && pushd protobuf-3.16.1 && \
- ./configure && sudo make && sudo checkinstall -y && \
- if [ ! -d $HOME/protobuf ]; then \
- mkdir -p $HOME/protobuf; \
- fi && \
- mv protobuf_3.16.1* $HOME/protobuf/ && popd; \
- fi && \
- sudo ldconfig && popd; \
+ if [ $PHASE = "build_checkstyle_protobuf" ]; then
+ sudo apt-get install -y libboost-all-dev libzookeeper-mt-dev libsasl2-dev cmake libcppunit-dev checkinstall &&
+ pushd .. &&
+ if [ -f $HOME/protobuf/protobuf_3.16.1* ]; then
+ sudo dpkg -i $HOME/protobuf/protobuf_3.16.1*;
+ else
+ wget https://github.com/protocolbuffers/protobuf/releases/download/v3.16.1/protobuf-java-3.16.1.zip &&
+ unzip protobuf-java-3.16.1.zip && pushd protobuf-3.16.1 &&
+ ./configure && sudo make && sudo checkinstall -y &&
+ if [ ! -d $HOME/protobuf ]; then
+ mkdir -p $HOME/protobuf;
+ fi &&
+ mv protobuf_3.16.1* $HOME/protobuf/ && popd;
+ fi &&
+ sudo ldconfig && popd;
fi
install:
# For tests phase runs unit tests
# For protobuf phase: builds Drill project, performs license checkstyle goal and regenerates Java and C++ Protobuf files
- |
- if [ $PHASE = "tests" ]; then \
- mvn install --batch-mode --no-transfer-progress \
- -DexcludedGroups="org.apache.drill.categories.SlowTest,org.apache.drill.categories.UnlikelyTest,org.apache.drill.categories.SecurityTest"; \
- elif [ $PHASE = "build_checkstyle_protobuf" ]; then \
- MAVEN_OPTS="-Xms1G -Xmx1G" mvn install --no-transfer-progress -Drat.skip=false -Dlicense.skip=false --batch-mode -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -DskipTests=true -Dmaven.javadoc.skip=true -Dmaven.source.skip=true && \
- pushd protocol && mvn process-sources -P proto-compile && popd && \
- mkdir contrib/native/client/build && pushd contrib/native/client/build && /usr/bin/cmake -G "Unix Makefiles" .. && make cpProtobufs && popd; \
+ if [ $PHASE = "tests" ]; then
+ echo Selecting parallel GC to minimise peak mem usage.
+ MAVEN_OPTS="-XX:+UseParallelGC"
Review comment:
Can we use the same GC as for regular Drill runs, so tests will be closer to the real envs? If not, why don't use UseParNewGC?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org