You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@creadur.apache.org by "Claudenw (via GitHub)" <gi...@apache.org> on 2024/05/04 12:06:11 UTC
[PR] RAT-372: Create archive processor [creadur-rat]
Claudenw opened a new pull request, #246:
URL: https://github.com/apache/creadur-rat/pull/246
Initial implementation
see https://issues.apache.org/jira/browse/RAT-372
See discussion: https://lists.apache.org/thread/gyp2tfsw3w66x0wy95tv1bcwb667h866
This pull is built upon the Tika changes in RAT-54
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-372: Create archive processor [creadur-rat]
Posted by "ottlinger (via GitHub)" <gi...@apache.org>.
ottlinger commented on code in PR #246:
URL: https://github.com/apache/creadur-rat/pull/246#discussion_r1597016972
##########
src/changes/changes.xml:
##########
@@ -107,12 +119,15 @@ https://maven.apache.org/plugins/maven-changes-plugin/xsd/changes-1.0.0.xsd
</action>
<action issue="RAT-366" type="update" dev="claudenw">
Switch to processing header matches in one call rather than line by line.
+
+ This change also resulted in the possibility of multiple licenses being detected and reported. forcing a change in the
+ XML ouptut. XML schema was developed for the output.
</action>
<action issue="RAT-333" type="fix" dev="claudenw">
Fix if --force option is used executable bit is not set properly on newly created/license-augmented file.
</action>
<action issue="RAT-345" type="update" dev="pottlinger" due-to="Niels Basjes">
- Update gitignore-reader from 1.4.0 to 1.5.1 to fetch changes resulting from fixes of RAT-362.
+ Update gitignore-reader from 1.4.0 to 1.5.1 to fetch changes resulting from fixes of RAT-362.
Review Comment:
What is the change here? as these entries are from older relases, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-372: Create archive processor [creadur-rat]
Posted by "Claudenw (via GitHub)" <gi...@apache.org>.
Claudenw commented on PR #246:
URL: https://github.com/apache/creadur-rat/pull/246#issuecomment-2105265426
The you can adjust the chatty level with the `--archive` settings. By default it is no more chatty than the earlier code.
I did change the XSLT so that the amount of space taken by the archive listing is reduced by 1/2, but if you ask for archive data to be added to the report, it has to get more chatty.
Is there a specific change you can recommend?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-372: Create archive processor [creadur-rat]
Posted by "Claudenw (via GitHub)" <gi...@apache.org>.
Claudenw commented on PR #246:
URL: https://github.com/apache/creadur-rat/pull/246#issuecomment-2101511004
For an interesting test run the following;
```
mvn clean install
java -jar apache-rat/target/apache-rat-0.17-SNAPSHOT.jar --archive PRESENCE .
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-372: Create archive processor [creadur-rat]
Posted by "ottlinger (via GitHub)" <gi...@apache.org>.
ottlinger commented on code in PR #246:
URL: https://github.com/apache/creadur-rat/pull/246#discussion_r1590322105
##########
apache-rat-core/src/main/java/org/apache/rat/Report.java:
##########
@@ -59,74 +65,166 @@
import org.apache.rat.walker.ArchiveWalker;
import org.apache.rat.walker.DirectoryWalker;
+import static java.lang.String.format;
+
/**
* The CLI based configuration object for report generation.
*/
public class Report {
+
+ private static final String[] NOTES = {
+ "Rat highlights possible issues.",
+ "Rat reports require interpretation.",
+ "Rat often requires some tuning before it runs well against a project.",
+ "Rat relies on heuristics: it may miss issues"
+ };
+
+ private static final String[] STYLE_SHEETS = {"plain-rat", "missing-headers", "unapproved-licenses"};
Review Comment:
Should we extract this into an enum -
apart from the text/CLI option value it could also contain the location of the corresponding XSLT and a description.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-372: Create archive processor [creadur-rat]
Posted by "ottlinger (via GitHub)" <gi...@apache.org>.
ottlinger commented on code in PR #246:
URL: https://github.com/apache/creadur-rat/pull/246#discussion_r1597227299
##########
src/changes/changes.xml:
##########
@@ -107,12 +119,15 @@ https://maven.apache.org/plugins/maven-changes-plugin/xsd/changes-1.0.0.xsd
</action>
<action issue="RAT-366" type="update" dev="claudenw">
Switch to processing header matches in one call rather than line by line.
+
+ This change also resulted in the possibility of multiple licenses being detected and reported. forcing a change in the
+ XML ouptut. XML schema was developed for the output.
</action>
<action issue="RAT-333" type="fix" dev="claudenw">
Fix if --force option is used executable bit is not set properly on newly created/license-augmented file.
</action>
<action issue="RAT-345" type="update" dev="pottlinger" due-to="Niels Basjes">
- Update gitignore-reader from 1.4.0 to 1.5.1 to fetch changes resulting from fixes of RAT-362.
+ Update gitignore-reader from 1.4.0 to 1.5.1 to fetch changes resulting from fixes of RAT-362.
Review Comment:
ok, go ahead :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-372: Create archive processor [creadur-rat]
Posted by "Claudenw (via GitHub)" <gi...@apache.org>.
Claudenw commented on code in PR #246:
URL: https://github.com/apache/creadur-rat/pull/246#discussion_r1597224503
##########
src/changes/changes.xml:
##########
@@ -107,12 +119,15 @@ https://maven.apache.org/plugins/maven-changes-plugin/xsd/changes-1.0.0.xsd
</action>
<action issue="RAT-366" type="update" dev="claudenw">
Switch to processing header matches in one call rather than line by line.
+
+ This change also resulted in the possibility of multiple licenses being detected and reported. forcing a change in the
+ XML ouptut. XML schema was developed for the output.
</action>
<action issue="RAT-333" type="fix" dev="claudenw">
Fix if --force option is used executable bit is not set properly on newly created/license-augmented file.
</action>
<action issue="RAT-345" type="update" dev="pottlinger" due-to="Niels Basjes">
- Update gitignore-reader from 1.4.0 to 1.5.1 to fetch changes resulting from fixes of RAT-362.
+ Update gitignore-reader from 1.4.0 to 1.5.1 to fetch changes resulting from fixes of RAT-362.
Review Comment:
The entries had some weird [NBSP] (non break space?) character, I replaced them with a space. I don't think it matters, but it made my IntelliJ editor much happier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-372: Create archive processor [creadur-rat]
Posted by "Claudenw (via GitHub)" <gi...@apache.org>.
Claudenw commented on code in PR #246:
URL: https://github.com/apache/creadur-rat/pull/246#discussion_r1590389356
##########
apache-rat-core/src/main/java/org/apache/rat/Report.java:
##########
@@ -59,74 +65,166 @@
import org.apache.rat.walker.ArchiveWalker;
import org.apache.rat.walker.DirectoryWalker;
+import static java.lang.String.format;
+
/**
* The CLI based configuration object for report generation.
*/
public class Report {
+
+ private static final String[] NOTES = {
+ "Rat highlights possible issues.",
+ "Rat reports require interpretation.",
+ "Rat often requires some tuning before it runs well against a project.",
+ "Rat relies on heuristics: it may miss issues"
+ };
+
+ private static final String[] STYLE_SHEETS = {"plain-rat", "missing-headers", "unapproved-licenses"};
Review Comment:
I made the change to an enum in the latest push. I also updated the site documentation to give background on how to write the XSLT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-372: Create archive processor [creadur-rat]
Posted by "ottlinger (via GitHub)" <gi...@apache.org>.
ottlinger commented on code in PR #246:
URL: https://github.com/apache/creadur-rat/pull/246#discussion_r1597010502
##########
apache-rat-core/src/main/java/org/apache/rat/document/impl/FileDocument.java:
##########
@@ -19,67 +19,71 @@
package org.apache.rat.document.impl;
import java.io.File;
-import java.io.FileInputStream;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.Reader;
-import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.SortedSet;
+import java.util.TreeSet;
import org.apache.rat.api.Document;
-import org.apache.rat.api.MetaData;
/**
- * Document wrapping a file of undetermined composition.
- *
+ * Document wrapping a File object
*/
-public class FileDocument implements Document {
+public class FileDocument extends Document {
+ /** the wrapped file */
private final File file;
- private final String name;
- private final MetaData metaData = new MetaData();
-
+
+ /**
+ * Creates a File document.
+ * @param file the file to wrap.
+ */
public FileDocument(final File file) {
- super();
+ super(normalizeFileName(file));
this.file = file;
- name = DocumentImplUtils.toName(file);
}
- public boolean isComposite() {
- return DocumentImplUtils.isZip(file);
+ /**
+ * Normalizes a file name. Accounts for Windows to Unix conversion.
+ * @param file
+ * @return
+ */
+ public final static String normalizeFileName(File file) {
+ String path = file.getPath();
+ return path.replace('\\', '/');
Review Comment:
Does this destroy escaped filenames with spaces in their file names? Just wondering
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-372: Create archive processor [creadur-rat]
Posted by "Claudenw (via GitHub)" <gi...@apache.org>.
Claudenw merged PR #246:
URL: https://github.com/apache/creadur-rat/pull/246
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-372: Create archive processor [creadur-rat]
Posted by "ottlinger (via GitHub)" <gi...@apache.org>.
ottlinger commented on PR #246:
URL: https://github.com/apache/creadur-rat/pull/246#issuecomment-2105267205
> The you can adjust the chatty level with the `--archive` settings. By default it is no more chatty than the earlier code.
>
> I did change the XSLT so that the amount of space taken by the archive listing is reduced by 1/2, but if you ask for archive data to be added to the report, it has to get more chatty.
>
> Is there a specific change you can recommend?
Not really, it basically was more or less a gut feeling when I saw the new report in a play project of mine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-372: Create archive processor [creadur-rat]
Posted by "Claudenw (via GitHub)" <gi...@apache.org>.
Claudenw commented on PR #246:
URL: https://github.com/apache/creadur-rat/pull/246#issuecomment-2094238845
This change also updated the Help routines and the CLI usage of commons-cli. New help screen now looks like:
```
usage: java -jar apache-rat/target/apache-rat-CURRENT-VERSION.jar [options] [DIR|TARBALL]
====== Available Options ======
-a (deprecated) Add the default license header to any file with an unknown license. Use '-A'
or ---addLicense instead.
-A,--addLicense Add the default license header to any file with an unknown license that is not in the
exclusion list. By default new files will be created with the license header, to force the
modification of existing files use the --force option.
--archive <ProcessingType> Specifies how ARCHIVE processing will be handled. (default is NOTIFICATION)
-c,--copyright <arg> The copyright message to use in the license headers, usually in the form of "Copyright 2008
Foo"
-d,--dir <DirOrArchive> (deprecated, use '--') Used to indicate source when using --exclude.
--dry-run If set do not update the files but generate the reports.
-e,--exclude <Expression> Excludes files matching wildcard <expression>. May be followed by multiple argumentsNote
that '--' or a following option is required when using this parameter.
-E,--exclude-file <FileOrURI> Excludes files matching regular expression in the input file.
-f,--force Forces any changes in files to be written directly to the source files (i.e. new files are
not created).
-h,--help Print help for the RAT command line interface and exit.
--licenses <FileOrURI> File names or URLs for license definitions
--list-families <LicenseFilter> List the defined license families (default is NONE). Valid options are: ALL, APPROVED, NONE
--list-licenses <LicenseFilter> List the defined licenses (default is NONE). Valid options are: ALL, APPROVED, NONE
--log-level <LogLevel> sets the log level.
--no-default-licenses Ignore default configuration. By default all approved default licenses are used
-o,--out <arg> Define the output file where to write a report to (default is System.out).
-s,--stylesheet <arg> XSLT stylesheet to use when creating the report. Not compatible with -x. Either an
external xsl file may be specified or one of the internal named sheets: plain-rat (default),
missing-headers, or unapproved-licenses
--scan-hidden-directories Scan hidden directories
-x,--xml Output the report in raw XML format. Not compatible with -s
====== Argument Types ======
<DirOrArchive>
A director or archive file to scan
<Expression>
A wildcard file matching pattern. example: *-test-*.txt
<FileOrURI>
A file name or URI
<LicenseFilter>
A defined filter for the licenses to include. Valid values: ALL, APPROVED, NONE.
<LogLevel>
The log level to use. Valid values DEBUG, INFO, WARN, ERROR, OFF.
<ProcessingType>
Specifies how to process file types. Valid values are:
NOTIFICATION: List archive as present
PRESENCE: List any licenses found
ABSENCE: List licenses found and any unknown licences
<StyleSheet>
Either an external xsl file may be one of the internal named sheets: missing-headers, plain-rat, unapproved-licenses.
====== Notes ======
1. Rat highlights possible issues.
2. Rat reports require interpretation.
3. Rat often requires some tuning before it runs well against a project.
4. Rat relies on heuristics: it may miss issues
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-372: Create archive processor [creadur-rat]
Posted by "ottlinger (via GitHub)" <gi...@apache.org>.
ottlinger commented on PR #246:
URL: https://github.com/apache/creadur-rat/pull/246#issuecomment-2094207899
Let's review this after https://github.com/apache/creadur-rat/pull/240 is merged into master - thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-372: Create archive processor [creadur-rat]
Posted by "Claudenw (via GitHub)" <gi...@apache.org>.
Claudenw commented on PR #246:
URL: https://github.com/apache/creadur-rat/pull/246#issuecomment-2101975568
Another interesting run
```
mvn clean install
mvn dependency:copy-dependencies -DoutputDirectory=/tmp/lib
java -jar apache-rat/target/apache-rat-0.17-SNAPSHOT.jar --archive PRESENCE /tmp/lib
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-372: Create archive processor [creadur-rat]
Posted by "Claudenw (via GitHub)" <gi...@apache.org>.
Claudenw commented on code in PR #246:
URL: https://github.com/apache/creadur-rat/pull/246#discussion_r1590328305
##########
apache-rat-core/src/main/java/org/apache/rat/Report.java:
##########
@@ -59,74 +65,166 @@
import org.apache.rat.walker.ArchiveWalker;
import org.apache.rat.walker.DirectoryWalker;
+import static java.lang.String.format;
+
/**
* The CLI based configuration object for report generation.
*/
public class Report {
+
+ private static final String[] NOTES = {
+ "Rat highlights possible issues.",
+ "Rat reports require interpretation.",
+ "Rat often requires some tuning before it runs well against a project.",
+ "Rat relies on heuristics: it may miss issues"
+ };
+
+ private static final String[] STYLE_SHEETS = {"plain-rat", "missing-headers", "unapproved-licenses"};
Review Comment:
We could do so and provide a descriptoin. We don't need the location as using `-s missing-headers` is all that is needed to specify the internal XSLT files. We might note where they are in comments so that developers can find them easily. Or point to them from documentation for XSLT development.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-372: Create archive processor [creadur-rat]
Posted by "ottlinger (via GitHub)" <gi...@apache.org>.
ottlinger commented on code in PR #246:
URL: https://github.com/apache/creadur-rat/pull/246#discussion_r1597227128
##########
apache-rat-core/src/main/java/org/apache/rat/document/impl/FileDocument.java:
##########
@@ -19,67 +19,71 @@
package org.apache.rat.document.impl;
import java.io.File;
-import java.io.FileInputStream;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.Reader;
-import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.SortedSet;
+import java.util.TreeSet;
import org.apache.rat.api.Document;
-import org.apache.rat.api.MetaData;
/**
- * Document wrapping a file of undetermined composition.
- *
+ * Document wrapping a File object
*/
-public class FileDocument implements Document {
+public class FileDocument extends Document {
+ /** the wrapped file */
private final File file;
- private final String name;
- private final MetaData metaData = new MetaData();
-
+
+ /**
+ * Creates a File document.
+ * @param file the file to wrap.
+ */
public FileDocument(final File file) {
- super();
+ super(normalizeFileName(file));
this.file = file;
- name = DocumentImplUtils.toName(file);
}
- public boolean isComposite() {
- return DocumentImplUtils.isZip(file);
+ /**
+ * Normalizes a file name. Accounts for Windows to Unix conversion.
+ * @param file
+ * @return
+ */
+ public final static String normalizeFileName(File file) {
+ String path = file.getPath();
+ return path.replace('\\', '/');
Review Comment:
Let's wait for any user feedback. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RAT-372: Create archive processor [creadur-rat]
Posted by "Claudenw (via GitHub)" <gi...@apache.org>.
Claudenw commented on code in PR #246:
URL: https://github.com/apache/creadur-rat/pull/246#discussion_r1597223544
##########
apache-rat-core/src/main/java/org/apache/rat/document/impl/FileDocument.java:
##########
@@ -19,67 +19,71 @@
package org.apache.rat.document.impl;
import java.io.File;
-import java.io.FileInputStream;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.Reader;
-import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.SortedSet;
+import java.util.TreeSet;
import org.apache.rat.api.Document;
-import org.apache.rat.api.MetaData;
/**
- * Document wrapping a file of undetermined composition.
- *
+ * Document wrapping a File object
*/
-public class FileDocument implements Document {
+public class FileDocument extends Document {
+ /** the wrapped file */
private final File file;
- private final String name;
- private final MetaData metaData = new MetaData();
-
+
+ /**
+ * Creates a File document.
+ * @param file the file to wrap.
+ */
public FileDocument(final File file) {
- super();
+ super(normalizeFileName(file));
this.file = file;
- name = DocumentImplUtils.toName(file);
}
- public boolean isComposite() {
- return DocumentImplUtils.isZip(file);
+ /**
+ * Normalizes a file name. Accounts for Windows to Unix conversion.
+ * @param file
+ * @return
+ */
+ public final static String normalizeFileName(File file) {
+ String path = file.getPath();
+ return path.replace('\\', '/');
Review Comment:
To be honest, I don't know. However, it is the same code that was found in a static class earlier. I just moved it here modified. I changed the name from `asName(File)` (?) to `normalizeFileName(File)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@creadur.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org