You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by GitBox <gi...@apache.org> on 2021/01/08 07:44:38 UTC
[GitHub] [nutch] lewismc opened a new pull request #561: NUTCH-2840 Fix 'report-vulnerabilities' ant target in build.xml
lewismc opened a new pull request #561:
URL: https://github.com/apache/nutch/pull/561
This PR addresses https://issues.apache.org/jira/browse/NUTCH-2840
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] sebastian-nagel commented on a change in pull request #561: NUTCH-2840 Fix 'report-vulnerabilities' ant target in build.xml
Posted by GitBox <gi...@apache.org>.
sebastian-nagel commented on a change in pull request #561:
URL: https://github.com/apache/nutch/pull/561#discussion_r566040843
##########
File path: build.xml
##########
@@ -646,24 +648,38 @@
</target>
<!-- Check dependencies for security vulnerabilities -->
- <!-- requires installation of OWASP dependency check tool, see -->
- <!-- https://jeremylong.github.io/DependencyCheck/dependency-check-ant/index.html -->
- <!-- get http://dl.bintray.com/jeremy-long/owasp/dependency-check-ant-3.3.2-release.zip -->
- <!-- and unzip in directory ./ivy/ -->
- <path id="dependency-check.path">
- <pathelement location="${dependency-check.home}/dependency-check-ant.jar"/>
- <fileset dir="${dependency-check.home}/lib" erroronmissingdir="false">
+ <target name="dependency-check-ant-download" description="--> download dependency-check-ant jar">
+ <available file="${dependency-check-ant.jar}" property="dependency-check-ant.jar.found"/>
+ <antcall target="dependency-check-ant-download-unchecked"/>
+ </target>
+
+ <target name="dependency-check-ant-download-unchecked" unless="dependency-check-ant.jar.found"
+ description="--> downloads the dependency-check-ant binary (dependency-check-ant-*.zip).">
+ <get src="https://github.com/jeremylong/DependencyCheck/releases/download/v${dependency-check-ant.version}/dependency-check-ant-${dependency-check-ant.version}-release.zip"
+ dest="${ivy.dir}/dependency-check-ant-${dependency-check-ant.version}-release.zip" usetimestamp="false" />
+
+ <unzip src="${ivy.dir}/dependency-check-ant-${dependency-check-ant.version}-release.zip"
+ dest="${ivy.dir}">
+ </unzip>
+
+ <delete file="${ivy.dir}/dependency-check-ant-${dependency-check-ant.version}-release.zip" />
+ </target>
+
+ <path id="dependency-check-ant.path">
+ <pathelement location="${dependency-check-ant.home}/dependency-check-ant.jar"/>
+ <fileset dir="${dependency-check-ant.home}/lib">
<include name="*.jar"/>
</fileset>
</path>
- <taskdef resource="dependency-check-taskdefs.properties" onerror="ignore">
- <classpath refid="dependency-check.path" />
+ <taskdef resource="dependency-check-taskdefs.properties">
Review comment:
Verified: moving `taskdef` few lines down, just after `<target name="report-vulnerabilities" ...` works.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] lewismc commented on pull request #561: NUTCH-2840 Fix 'report-vulnerabilities' ant target in build.xml
Posted by GitBox <gi...@apache.org>.
lewismc commented on pull request #561:
URL: https://github.com/apache/nutch/pull/561#issuecomment-758867279
Is anyone else able to test and see if they can reproduce error's report by Seb? Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] lewismc commented on pull request #561: NUTCH-2840 Fix 'report-vulnerabilities' ant target in build.xml
Posted by GitBox <gi...@apache.org>.
lewismc commented on pull request #561:
URL: https://github.com/apache/nutch/pull/561#issuecomment-757014801
OK folks, I finally stabilized this build. Some ant magic was required. Please try it out and let me know what you think.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] sebastian-nagel commented on pull request #561: NUTCH-2840 Fix 'report-vulnerabilities' ant target in build.xml
Posted by GitBox <gi...@apache.org>.
sebastian-nagel commented on pull request #561:
URL: https://github.com/apache/nutch/pull/561#issuecomment-769041148
Could also upgrade to DependencyCheck 6.1.0 ([released yesterday](https://github.com/jeremylong/DependencyCheck/releases)).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] sebastian-nagel commented on pull request #561: NUTCH-2840 Fix 'report-vulnerabilities' ant target in build.xml
Posted by GitBox <gi...@apache.org>.
sebastian-nagel commented on pull request #561:
URL: https://github.com/apache/nutch/pull/561#issuecomment-768966714
> I am also experiencing these issues... I'll look into them
Thanks, @lewismc!
> [junit] Test org.apache.nutch.protocol.httpclient.TestProtocolHttpClient FAILED
The HTTP protocol unit tests launch a Jetty server to handle HTTP requests. The Jetty instances listen on hard-coded ports, and sometimes opening a port fails because it is already occupied.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] lewismc commented on a change in pull request #561: NUTCH-2840 Fix 'report-vulnerabilities' ant target in build.xml
Posted by GitBox <gi...@apache.org>.
lewismc commented on a change in pull request #561:
URL: https://github.com/apache/nutch/pull/561#discussion_r556001913
##########
File path: build.xml
##########
@@ -639,24 +641,38 @@
</target>
<!-- Check dependencies for security vulnerabilities -->
- <!-- requires installation of OWASP dependency check tool, see -->
- <!-- https://jeremylong.github.io/DependencyCheck/dependency-check-ant/index.html -->
- <!-- get http://dl.bintray.com/jeremy-long/owasp/dependency-check-ant-3.3.2-release.zip -->
- <!-- and unzip in directory ./ivy/ -->
- <path id="dependency-check.path">
- <pathelement location="${dependency-check.home}/dependency-check-ant.jar"/>
- <fileset dir="${dependency-check.home}/lib" erroronmissingdir="false">
+ <target name="dependency-check-ant-download" description="--> download dependency-check-ant jar">
+ <available file="${dependency-check-ant.jar}" property="dependency-check-ant.jar.found"/>
+ <antcall target="dependency-check-ant-download-unchecked"/>
+ </target>
+
+ <target name="dependency-check-ant-download-unchecked" unless="dependency-check-ant.jar.found"
+ description="--> downloads the dependency-check-ant binary (dependency-check-ant-*.zip).">
+ <get src="https://github.com/jeremylong/DependencyCheck/releases/download/v${dependency-check-ant.version}/dependency-check-ant-${dependency-check-ant.version}-release.zip"
+ dest="${ivy.dir}/dependency-check-ant-${dependency-check-ant.version}-release.zip" usetimestamp="false" />
+
+ <unzip src="${dependency-check-ant.home}/dependency-check-ant-${dependency-check-ant.version}-release.zip"
Review comment:
Good catch. Thank you
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] lewismc commented on pull request #561: NUTCH-2840 Fix 'report-vulnerabilities' ant target in build.xml
Posted by GitBox <gi...@apache.org>.
lewismc commented on pull request #561:
URL: https://github.com/apache/nutch/pull/561#issuecomment-757014801
OK folks, I finally stabilized this build. Some ant magic was required. Please try it out and let me know what you think.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] lewismc commented on pull request #561: NUTCH-2840 Fix 'report-vulnerabilities' ant target in build.xml
Posted by GitBox <gi...@apache.org>.
lewismc commented on pull request #561:
URL: https://github.com/apache/nutch/pull/561#issuecomment-768743046
@sebastian-nagel
# 1 has been fixed. I just provided thew static string `apache-nutch`
Regarding # 2. I cleared ~/.ivy2/cache and made a clean checkout of nutch master branch. I then ran `ant check-vulnerabilities` and
Also, it appears that the Github build machine ran into the following unit test failure
```
[junit] Tests run: 7, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 73.317 sec
[junit] Test org.apache.nutch.protocol.httpclient.TestProtocolHttpClient FAILED
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] lewismc merged pull request #561: NUTCH-2840 Fix 'report-vulnerabilities' ant target in build.xml
Posted by GitBox <gi...@apache.org>.
lewismc merged pull request #561:
URL: https://github.com/apache/nutch/pull/561
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] sebastian-nagel commented on a change in pull request #561: NUTCH-2840 Fix 'report-vulnerabilities' ant target in build.xml
Posted by GitBox <gi...@apache.org>.
sebastian-nagel commented on a change in pull request #561:
URL: https://github.com/apache/nutch/pull/561#discussion_r566035975
##########
File path: build.xml
##########
@@ -646,24 +648,38 @@
</target>
<!-- Check dependencies for security vulnerabilities -->
- <!-- requires installation of OWASP dependency check tool, see -->
- <!-- https://jeremylong.github.io/DependencyCheck/dependency-check-ant/index.html -->
- <!-- get http://dl.bintray.com/jeremy-long/owasp/dependency-check-ant-3.3.2-release.zip -->
- <!-- and unzip in directory ./ivy/ -->
- <path id="dependency-check.path">
- <pathelement location="${dependency-check.home}/dependency-check-ant.jar"/>
- <fileset dir="${dependency-check.home}/lib" erroronmissingdir="false">
+ <target name="dependency-check-ant-download" description="--> download dependency-check-ant jar">
+ <available file="${dependency-check-ant.jar}" property="dependency-check-ant.jar.found"/>
+ <antcall target="dependency-check-ant-download-unchecked"/>
+ </target>
+
+ <target name="dependency-check-ant-download-unchecked" unless="dependency-check-ant.jar.found"
+ description="--> downloads the dependency-check-ant binary (dependency-check-ant-*.zip).">
+ <get src="https://github.com/jeremylong/DependencyCheck/releases/download/v${dependency-check-ant.version}/dependency-check-ant-${dependency-check-ant.version}-release.zip"
+ dest="${ivy.dir}/dependency-check-ant-${dependency-check-ant.version}-release.zip" usetimestamp="false" />
+
+ <unzip src="${ivy.dir}/dependency-check-ant-${dependency-check-ant.version}-release.zip"
+ dest="${ivy.dir}">
+ </unzip>
+
+ <delete file="${ivy.dir}/dependency-check-ant-${dependency-check-ant.version}-release.zip" />
+ </target>
+
+ <path id="dependency-check-ant.path">
+ <pathelement location="${dependency-check-ant.home}/dependency-check-ant.jar"/>
+ <fileset dir="${dependency-check-ant.home}/lib">
<include name="*.jar"/>
</fileset>
</path>
- <taskdef resource="dependency-check-taskdefs.properties" onerror="ignore">
- <classpath refid="dependency-check.path" />
+ <taskdef resource="dependency-check-taskdefs.properties">
Review comment:
While working on NUTCH-2819 (#565) I've also run into the `Cause: The name is undefined.` error. Moving the task definition down into the target it belongs to, resolved this error.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] sebastian-nagel commented on pull request #561: NUTCH-2840 Fix 'report-vulnerabilities' ant target in build.xml
Posted by GitBox <gi...@apache.org>.
sebastian-nagel commented on pull request #561:
URL: https://github.com/apache/nutch/pull/561#issuecomment-769028827
Regarding the second error (`NoSuchMethodError`): I was able to solve it by cleaning up my `~/.ant/lib/` folder. The class `com.google.common.io.Files` was also included in some of the jars there, with a version incompatible to that required by dependency-check tool.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] lewismc commented on a change in pull request #561: NUTCH-2840 Fix 'report-vulnerabilities' ant target in build.xml
Posted by GitBox <gi...@apache.org>.
lewismc commented on a change in pull request #561:
URL: https://github.com/apache/nutch/pull/561#discussion_r566296857
##########
File path: build.xml
##########
@@ -646,24 +648,38 @@
</target>
<!-- Check dependencies for security vulnerabilities -->
- <!-- requires installation of OWASP dependency check tool, see -->
- <!-- https://jeremylong.github.io/DependencyCheck/dependency-check-ant/index.html -->
- <!-- get http://dl.bintray.com/jeremy-long/owasp/dependency-check-ant-3.3.2-release.zip -->
- <!-- and unzip in directory ./ivy/ -->
- <path id="dependency-check.path">
- <pathelement location="${dependency-check.home}/dependency-check-ant.jar"/>
- <fileset dir="${dependency-check.home}/lib" erroronmissingdir="false">
+ <target name="dependency-check-ant-download" description="--> download dependency-check-ant jar">
+ <available file="${dependency-check-ant.jar}" property="dependency-check-ant.jar.found"/>
+ <antcall target="dependency-check-ant-download-unchecked"/>
+ </target>
+
+ <target name="dependency-check-ant-download-unchecked" unless="dependency-check-ant.jar.found"
+ description="--> downloads the dependency-check-ant binary (dependency-check-ant-*.zip).">
+ <get src="https://github.com/jeremylong/DependencyCheck/releases/download/v${dependency-check-ant.version}/dependency-check-ant-${dependency-check-ant.version}-release.zip"
+ dest="${ivy.dir}/dependency-check-ant-${dependency-check-ant.version}-release.zip" usetimestamp="false" />
+
+ <unzip src="${ivy.dir}/dependency-check-ant-${dependency-check-ant.version}-release.zip"
+ dest="${ivy.dir}">
+ </unzip>
+
+ <delete file="${ivy.dir}/dependency-check-ant-${dependency-check-ant.version}-release.zip" />
+ </target>
+
+ <path id="dependency-check-ant.path">
+ <pathelement location="${dependency-check-ant.home}/dependency-check-ant.jar"/>
+ <fileset dir="${dependency-check-ant.home}/lib">
<include name="*.jar"/>
</fileset>
</path>
- <taskdef resource="dependency-check-taskdefs.properties" onerror="ignore">
- <classpath refid="dependency-check.path" />
+ <taskdef resource="dependency-check-taskdefs.properties">
Review comment:
Thanks @sebastian-nagel
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] sebastian-nagel commented on a change in pull request #561: NUTCH-2840 Fix 'report-vulnerabilities' ant target in build.xml
Posted by GitBox <gi...@apache.org>.
sebastian-nagel commented on a change in pull request #561:
URL: https://github.com/apache/nutch/pull/561#discussion_r554980570
##########
File path: build.xml
##########
@@ -639,24 +641,38 @@
</target>
<!-- Check dependencies for security vulnerabilities -->
- <!-- requires installation of OWASP dependency check tool, see -->
- <!-- https://jeremylong.github.io/DependencyCheck/dependency-check-ant/index.html -->
- <!-- get http://dl.bintray.com/jeremy-long/owasp/dependency-check-ant-3.3.2-release.zip -->
- <!-- and unzip in directory ./ivy/ -->
- <path id="dependency-check.path">
- <pathelement location="${dependency-check.home}/dependency-check-ant.jar"/>
- <fileset dir="${dependency-check.home}/lib" erroronmissingdir="false">
+ <target name="dependency-check-ant-download" description="--> download dependency-check-ant jar">
+ <available file="${dependency-check-ant.jar}" property="dependency-check-ant.jar.found"/>
+ <antcall target="dependency-check-ant-download-unchecked"/>
+ </target>
+
+ <target name="dependency-check-ant-download-unchecked" unless="dependency-check-ant.jar.found"
+ description="--> downloads the dependency-check-ant binary (dependency-check-ant-*.zip).">
+ <get src="https://github.com/jeremylong/DependencyCheck/releases/download/v${dependency-check-ant.version}/dependency-check-ant-${dependency-check-ant.version}-release.zip"
+ dest="${ivy.dir}/dependency-check-ant-${dependency-check-ant.version}-release.zip" usetimestamp="false" />
+
+ <unzip src="${dependency-check-ant.home}/dependency-check-ant-${dependency-check-ant.version}-release.zip"
Review comment:
The parameter `src` should be `${ivy.dir}/dependency-check-ant-${dependency-check-ant.version}-release.zip` - same as download destination and file to be deleted later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] lewismc edited a comment on pull request #561: NUTCH-2840 Fix 'report-vulnerabilities' ant target in build.xml
Posted by GitBox <gi...@apache.org>.
lewismc edited a comment on pull request #561:
URL: https://github.com/apache/nutch/pull/561#issuecomment-768743046
@sebastian-nagel
I am also experiencing these issues... I'll look into them
Also, it appears that the Github build machine ran into the following unit test failure
```
[junit] Tests run: 7, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 73.317 sec
[junit] Test org.apache.nutch.protocol.httpclient.TestProtocolHttpClient FAILED
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org