You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@sling.apache.org by ro...@apache.org on 2019/05/08 16:16:04 UTC
[sling-whiteboard] branch master updated (a1f9157 -> 51f552e)
This is an automated email from the ASF dual-hosted git repository.
rombert pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/sling-whiteboard.git.
from a1f9157 Merge branch 'master' of https://github.com/apache/sling-whiteboard
new 78fce02 Add basic tool to find source artifacts on Maven central
new 49c79ae Fix build and ignore out directory
new 29e35cf Remove unused guice dependency
new 2771821 Improvements:
new 64fbcfe Updated README
new 08c6db2 Remove unneeded indexers.
new 27d8acc Formatting
new e7469e3 Adjust artifactId and name
new 51f552e Renamed folder
The 9 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
maven-central-source-reporter/.gitignore | 1 +
maven-central-source-reporter/README.md | 16 ++
maven-central-source-reporter/pom.xml | 111 ++++++++++++
.../sling/tooling/msra/impl/IndexQuerier.java | 198 +++++++++++++++++++++
.../org/apache/sling/tooling/msra/impl/Main.java | 56 +++---
5 files changed, 359 insertions(+), 23 deletions(-)
create mode 100644 maven-central-source-reporter/.gitignore
create mode 100644 maven-central-source-reporter/README.md
create mode 100644 maven-central-source-reporter/pom.xml
create mode 100644 maven-central-source-reporter/src/main/java/org/apache/sling/tooling/msra/impl/IndexQuerier.java
copy mvresourceprovider/src/main/java/org/apache/sling/mvresource/impl/MvSession.java => maven-central-source-reporter/src/main/java/org/apache/sling/tooling/msra/impl/Main.java (66%)
[sling-whiteboard] 03/09: Remove unused guice dependency
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rombert pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/sling-whiteboard.git
commit 29e35cf9a868dc3cfadb84d357d78ca0798f9a97
Author: Robert Munteanu <ro...@apache.org>
AuthorDate: Mon May 6 16:36:43 2019 +0200
Remove unused guice dependency
---
maven-sources-report-aggregator/pom.xml | 6 +-----
1 file changed, 1 insertion(+), 5 deletions(-)
diff --git a/maven-sources-report-aggregator/pom.xml b/maven-sources-report-aggregator/pom.xml
index 1b91e4a..a3ebad4 100644
--- a/maven-sources-report-aggregator/pom.xml
+++ b/maven-sources-report-aggregator/pom.xml
@@ -32,6 +32,7 @@
<name>Apache Sling - Maven Source Artifacts Report Aggregator</name>
<dependencies>
+ <!-- Main dependency -->
<dependency>
<groupId>org.apache.maven.indexer</groupId>
<artifactId>indexer-core</artifactId>
@@ -70,11 +71,6 @@
<artifactId>org.eclipse.sisu.plexus</artifactId>
<version>0.3.3</version>
</dependency>
- <dependency>
- <groupId>com.google.inject</groupId>
- <artifactId>guice</artifactId>
- <version>4.1.0</version>
- </dependency>
</dependencies>
<build>
[sling-whiteboard] 01/09: Add basic tool to find source artifacts
on Maven central
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rombert pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/sling-whiteboard.git
commit 78fce02a12498345c0e4b83175984e67d98debf3
Author: Robert Munteanu <ro...@apache.org>
AuthorDate: Mon May 6 16:16:14 2019 +0200
Add basic tool to find source artifacts on Maven central
---
maven-sources-report-aggregator/pom.xml | 87 ++++++++
.../org/apache/sling/tooling/msra/impl/Main.java | 233 +++++++++++++++++++++
2 files changed, 320 insertions(+)
diff --git a/maven-sources-report-aggregator/pom.xml b/maven-sources-report-aggregator/pom.xml
new file mode 100644
index 0000000..3399ccd
--- /dev/null
+++ b/maven-sources-report-aggregator/pom.xml
@@ -0,0 +1,87 @@
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.sling</groupId>
+ <artifactId>sling</artifactId>
+ <version>35</version>
+ <relativePath />
+ </parent>
+
+ <artifactId>org.apache.sling.tooling.maven-sources-report-aggregator</artifactId>
+ <version>0.0.1-SNAPSHOT</version>
+ <name>Apache Sling - Maven Source Artifacts Report Aggregator</name>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.maven.indexer</groupId>
+ <artifactId>indexer-core</artifactId>
+ <version>6.0.0</version>
+ </dependency>
+
+ <!-- Ensure Slf4j ends up in the fat jar -->
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-simple</artifactId>
+ <scope>compile</scope>
+ </dependency>
+
+ <!-- ensure javax.inject ends up in the fat jar -->
+ <dependency>
+ <groupId>javax.inject</groupId>
+ <artifactId>javax.inject</artifactId>
+ <scope>compile</scope>
+ </dependency>
+
+ <!-- For ResourceFetcher implementation, if used -->
+ <dependency>
+ <groupId>org.apache.maven.wagon</groupId>
+ <artifactId>wagon-http-lightweight</artifactId>
+ <version>2.12</version>
+ </dependency>
+
+ <!-- Runtime: DI, but using Plexus Shim as we use Wagon -->
+ <dependency>
+ <groupId>org.eclipse.sisu</groupId>
+ <artifactId>org.eclipse.sisu.plexus</artifactId>
+ <version>0.3.3</version>
+ </dependency>
+ <dependency>
+ <groupId>com.google.inject</groupId>
+ <artifactId>guice</artifactId>
+ <version>4.1.0</version>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <plugins>
+ <plugin>
+ <artifactId>maven-assembly-plugin</artifactId>
+ <configuration>
+ <archive>
+ <manifest>
+ <mainClass>org.apache.sling.tooling.msra.impl.Main</mainClass>
+ </manifest>
+ </archive>
+ <descriptorRefs>
+ <descriptorRef>jar-with-dependencies</descriptorRef>
+ </descriptorRefs>
+ </configuration>
+ <executions>
+ <execution>
+ <phase>package</phase>
+ <goals>
+ <goal>single</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+</project>
\ No newline at end of file
diff --git a/maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/Main.java b/maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/Main.java
new file mode 100644
index 0000000..ce9d4a4
--- /dev/null
+++ b/maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/Main.java
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.sling.tooling.msra.impl;
+
+import java.io.BufferedOutputStream;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+import java.util.Objects;
+
+import org.apache.lucene.search.BooleanClause.Occur;
+import org.apache.lucene.search.BooleanQuery;
+import org.apache.lucene.search.Query;
+import org.apache.maven.index.ArtifactInfo;
+import org.apache.maven.index.ArtifactInfoGroup;
+import org.apache.maven.index.GroupedSearchRequest;
+import org.apache.maven.index.GroupedSearchResponse;
+import org.apache.maven.index.Indexer;
+import org.apache.maven.index.MAVEN;
+import org.apache.maven.index.context.IndexCreator;
+import org.apache.maven.index.context.IndexingContext;
+import org.apache.maven.index.expr.SourcedSearchExpression;
+import org.apache.maven.index.search.grouping.GAGrouping;
+import org.apache.maven.index.updater.IndexUpdateRequest;
+import org.apache.maven.index.updater.IndexUpdateResult;
+import org.apache.maven.index.updater.IndexUpdater;
+import org.apache.maven.index.updater.ResourceFetcher;
+import org.apache.maven.index.updater.WagonHelper;
+import org.apache.maven.wagon.Wagon;
+import org.apache.maven.wagon.events.TransferEvent;
+import org.apache.maven.wagon.events.TransferListener;
+import org.apache.maven.wagon.observers.AbstractTransferListener;
+import org.codehaus.plexus.DefaultContainerConfiguration;
+import org.codehaus.plexus.DefaultPlexusContainer;
+import org.codehaus.plexus.PlexusConstants;
+import org.codehaus.plexus.PlexusContainer;
+import org.codehaus.plexus.PlexusContainerException;
+import org.codehaus.plexus.component.repository.exception.ComponentLookupException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class Main {
+
+ public static void main(String[] args) throws Exception {
+
+ Main main = new Main("org.apache.sling*");
+ main.loadIndex();
+ main.querySlingApiArtifacts();
+ }
+
+ private static final File OUT_DIR = new File("out");
+
+ private final PlexusContainer plexusContainer;
+
+ private final Indexer indexer;
+
+ private final IndexUpdater indexUpdater;
+
+ private final Wagon httpWagon;
+
+ private IndexingContext centralContext;
+
+ private final Logger logger = LoggerFactory.getLogger(getClass());
+
+ private final String queryGroupId;
+
+ public Main(String queryGroupId) throws PlexusContainerException, ComponentLookupException {
+ // here we create Plexus container, the Maven default IoC container
+ // Plexus falls outside of MI scope, just accept the fact that
+ // MI is a Plexus component ;)
+ // If needed more info, ask on Maven Users list or Plexus Users list
+ // google is your friend!
+ final DefaultContainerConfiguration config = new DefaultContainerConfiguration();
+ config.setClassPathScanning(PlexusConstants.SCANNING_INDEX);
+ this.plexusContainer = new DefaultPlexusContainer(config);
+
+ // lookup the indexer components from plexus
+ this.indexer = plexusContainer.lookup(Indexer.class);
+ this.indexUpdater = plexusContainer.lookup(IndexUpdater.class);
+ // lookup wagon used to remotely fetch index
+ this.httpWagon = plexusContainer.lookup(Wagon.class, "https");
+ this.queryGroupId = queryGroupId;
+
+ }
+
+ public void loadIndex() throws ComponentLookupException, IOException {
+ // Files where local cache is (if any) and Lucene Index should be located
+ File centralLocalCache = new File(OUT_DIR, "central-cache");
+ File centralIndexDir = new File(OUT_DIR, "central-index");
+
+ // Creators we want to use (search for fields it defines)
+ List<IndexCreator> indexers = new ArrayList<>();
+ indexers.add(plexusContainer.lookup(IndexCreator.class, "min"));
+ indexers.add(plexusContainer.lookup(IndexCreator.class, "jarContent"));
+ indexers.add(plexusContainer.lookup(IndexCreator.class, "maven-plugin"));
+
+ // Create context for central repository index
+ centralContext = indexer.createIndexingContext("central-context", "central", centralLocalCache, centralIndexDir,
+ "https://repo1.maven.org/maven2", null, true, true, indexers);
+
+ // Update the index (incremental update will happen if this is not 1st run and
+ // files are not deleted)
+ // This whole block below should not be executed on every app start, but rather
+ // controlled by some configuration
+ // since this block will always emit at least one HTTP GET. Central indexes are
+ // updated once a week, but
+ // other index sources might have different index publishing frequency.
+ // Preferred frequency is once a week.
+ logger.info("Updating Index. This might take a while on first run, so please be patient.");
+ // Create ResourceFetcher implementation to be used with IndexUpdateRequest
+ // Here, we use Wagon based one as shorthand, but all we need is a
+ // ResourceFetcher implementation
+ TransferListener listener = new AbstractTransferListener() {
+ @Override
+ public void transferStarted(TransferEvent transferEvent) {
+ logger.info("Downloading {}", transferEvent.getResource().getName());
+ }
+
+ @Override
+ public void transferCompleted(TransferEvent transferEvent) {
+ logger.info("Done downloading {}", transferEvent.getResource().getName());
+ }
+ };
+ ResourceFetcher resourceFetcher = new WagonHelper.WagonFetcher(httpWagon, listener, null, null);
+
+ Date centralContextCurrentTimestamp = centralContext.getTimestamp();
+ IndexUpdateRequest updateRequest = new IndexUpdateRequest(centralContext, resourceFetcher);
+ IndexUpdateResult updateResult = indexUpdater.fetchAndUpdateIndex(updateRequest);
+ if (updateResult.isFullUpdate()) {
+ logger.info("Full update happened!");
+ } else if (updateResult.getTimestamp().equals(centralContextCurrentTimestamp)) {
+ logger.info("No update needed, index is up to date!");
+ } else {
+ logger.info("Incremental update happened, change covered {} - {} period.",
+ centralContextCurrentTimestamp , updateResult.getTimestamp());
+ }
+ }
+
+ public void querySlingApiArtifacts() throws IOException {
+
+ Query groupId = indexer.constructQuery(MAVEN.GROUP_ID, new SourcedSearchExpression(queryGroupId));
+ Query packaging = indexer.constructQuery(MAVEN.PACKAGING, new SourcedSearchExpression("pom"));
+
+ BooleanQuery bq = new BooleanQuery.Builder()
+ .add(groupId, Occur.MUST)
+ .add(packaging, Occur.MUST_NOT)
+ .build();
+
+ searchAndDump( indexer, "all " + queryGroupId + " artifacts", bq );
+ }
+
+ public void searchAndDump(Indexer nexusIndexer, String descr, Query q) throws IOException {
+ logger.info("Searching for {}", descr);
+
+ GroupedSearchResponse response = nexusIndexer
+ .searchGrouped(new GroupedSearchRequest(q, new GAGrouping(), centralContext));
+
+ try ( FileOutputStream fos = new FileOutputStream(new File(OUT_DIR, "results.csv"));
+ BufferedOutputStream bos = new BufferedOutputStream(fos);
+ OutputStreamWriter w = new OutputStreamWriter(bos)) {
+
+ response.getResults().values()
+ .stream()
+ .map( Main::extract )
+ .filter( Objects::nonNull )
+ .forEach( aws -> writeLine(w, aws));
+
+ logger.info("Total hits: {}", response.getTotalHitsCount());
+ }
+
+ logger.info("Data written under {}", OUT_DIR);
+ }
+
+ private void writeLine(OutputStreamWriter w, ArtifactWithSource aws) {
+ try {
+ w.write(aws.main.getGroupId() + ',' +
+ aws.main.getArtifactId()+ ',' +
+ aws.main.getVersion()+ ',' +
+ (aws.source != null ? '1' : '0') + '\n');
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private static ArtifactWithSource extract(ArtifactInfoGroup group) {
+
+ ArtifactWithSource aws = new ArtifactWithSource();
+
+ aws.main = group.getArtifactInfos()
+ .stream()
+ .filter( a -> a.getClassifier() == null)
+ .findFirst()
+ .orElse(null);
+
+ if ( aws.main == null )
+ return null;
+
+ // TODO - need constant for 'sources'
+ aws.source = group.getArtifactInfos().stream()
+ .filter( a -> a.getVersion().equals(aws.main.getVersion()) && "sources".equals(a.getClassifier()) )
+ .findFirst()
+ .orElse(null);
+
+ return aws;
+ }
+
+ static class ArtifactWithSource {
+
+ private ArtifactInfo main;
+
+ private ArtifactInfo source;
+ }
+}
[sling-whiteboard] 04/09: Improvements:
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rombert pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/sling-whiteboard.git
commit 277182112fd1c99aceb325240340548a288d8cd3
Author: Robert Munteanu <ro...@apache.org>
AuthorDate: Mon May 6 16:46:43 2019 +0200
Improvements:
- Move the bulk of the code to an IndexQuerier class
- Use the available API to check if sources are present
---
.../msra/impl/{Main.java => IndexQuerier.java} | 110 ++++-------
.../org/apache/sling/tooling/msra/impl/Main.java | 210 +--------------------
2 files changed, 38 insertions(+), 282 deletions(-)
diff --git a/maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/Main.java b/maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/IndexQuerier.java
similarity index 72%
copy from maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/Main.java
copy to maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/IndexQuerier.java
index 2165203..d5245c0 100644
--- a/maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/Main.java
+++ b/maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/IndexQuerier.java
@@ -18,6 +18,8 @@
*/
package org.apache.sling.tooling.msra.impl;
+import static org.apache.maven.index.ArtifactAvailability.PRESENT;
+
import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileOutputStream;
@@ -28,9 +30,9 @@ import java.util.Date;
import java.util.List;
import java.util.Objects;
-import org.apache.lucene.search.BooleanClause.Occur;
import org.apache.lucene.search.BooleanQuery;
import org.apache.lucene.search.Query;
+import org.apache.lucene.search.BooleanClause.Occur;
import org.apache.maven.index.ArtifactInfo;
import org.apache.maven.index.ArtifactInfoGroup;
import org.apache.maven.index.GroupedSearchRequest;
@@ -59,19 +61,17 @@ import org.codehaus.plexus.component.repository.exception.ComponentLookupExcepti
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-public class Main {
+public class IndexQuerier {
- public static void main(String[] args) throws Exception {
-
- if ( args.length == 0 ) {
- throw new IllegalArgumentException("Please specify groupId to look for, e.g. org.apache*");
- }
-
- Main main = new Main(args[0]);
- main.loadIndex();
- main.querySlingApiArtifacts();
+ private static ArtifactInfo extract(ArtifactInfoGroup group) {
+
+ return group.getArtifactInfos()
+ .stream()
+ .filter(a -> a.getClassifier() == null)
+ .findFirst()
+ .orElse(null);
}
-
+
private static final File OUT_DIR = new File("out");
private final PlexusContainer plexusContainer;
@@ -83,19 +83,15 @@ public class Main {
private final Wagon httpWagon;
private IndexingContext centralContext;
-
+
private final Logger logger = LoggerFactory.getLogger(getClass());
- private final String queryGroupId;
- public Main(String queryGroupId) throws PlexusContainerException, ComponentLookupException {
- // here we create Plexus container, the Maven default IoC container
- // Plexus falls outside of MI scope, just accept the fact that
- // MI is a Plexus component ;)
- // If needed more info, ask on Maven Users list or Plexus Users list
- // google is your friend!
- final DefaultContainerConfiguration config = new DefaultContainerConfiguration();
+ public IndexQuerier() throws PlexusContainerException, ComponentLookupException {
+
+ DefaultContainerConfiguration config = new DefaultContainerConfiguration();
config.setClassPathScanning(PlexusConstants.SCANNING_INDEX);
+
this.plexusContainer = new DefaultPlexusContainer(config);
// lookup the indexer components from plexus
@@ -103,8 +99,6 @@ public class Main {
this.indexUpdater = plexusContainer.lookup(IndexUpdater.class);
// lookup wagon used to remotely fetch index
this.httpWagon = plexusContainer.lookup(Wagon.class, "https");
- this.queryGroupId = queryGroupId;
-
}
public void loadIndex() throws ComponentLookupException, IOException {
@@ -155,22 +149,19 @@ public class Main {
} else if (updateResult.getTimestamp().equals(centralContextCurrentTimestamp)) {
logger.info("No update needed, index is up to date!");
} else {
- logger.info("Incremental update happened, change covered {} - {} period.",
- centralContextCurrentTimestamp , updateResult.getTimestamp());
+ logger.info("Incremental update happened, change covered {} - {} period.", centralContextCurrentTimestamp,
+ updateResult.getTimestamp());
}
}
-
- public void querySlingApiArtifacts() throws IOException {
-
+
+ public void querySourceArtifacts(String queryGroupId) throws IOException {
+
Query groupId = indexer.constructQuery(MAVEN.GROUP_ID, new SourcedSearchExpression(queryGroupId));
Query packaging = indexer.constructQuery(MAVEN.PACKAGING, new SourcedSearchExpression("pom"));
- BooleanQuery bq = new BooleanQuery.Builder()
- .add(groupId, Occur.MUST)
- .add(packaging, Occur.MUST_NOT)
- .build();
+ BooleanQuery bq = new BooleanQuery.Builder().add(groupId, Occur.MUST).add(packaging, Occur.MUST_NOT).build();
- searchAndDump( indexer, "all " + queryGroupId + " artifacts", bq );
+ searchAndDump(indexer, "all " + queryGroupId + " artifacts", bq);
}
public void searchAndDump(Indexer nexusIndexer, String descr, Query q) throws IOException {
@@ -179,59 +170,28 @@ public class Main {
GroupedSearchResponse response = nexusIndexer
.searchGrouped(new GroupedSearchRequest(q, new GAGrouping(), centralContext));
- try ( FileOutputStream fos = new FileOutputStream(new File(OUT_DIR, "results.csv"));
- BufferedOutputStream bos = new BufferedOutputStream(fos);
+ try (FileOutputStream fos = new FileOutputStream(new File(OUT_DIR, "results.csv"));
+ BufferedOutputStream bos = new BufferedOutputStream(fos);
OutputStreamWriter w = new OutputStreamWriter(bos)) {
-
+
response.getResults().values()
.stream()
- .map( Main::extract )
- .filter( Objects::nonNull )
- .forEach( aws -> writeLine(w, aws));
-
+ .map(IndexQuerier::extract)
+ .filter(Objects::nonNull)
+ .forEach(ai -> writeLine(w, ai));
+
logger.info("Total hits: {}", response.getTotalHitsCount());
}
-
+
logger.info("Data written under {}", OUT_DIR);
}
- private void writeLine(OutputStreamWriter w, ArtifactWithSource aws) {
+ private void writeLine(OutputStreamWriter w, ArtifactInfo aws) {
try {
- w.write(aws.main.getGroupId() + ',' +
- aws.main.getArtifactId()+ ',' +
- aws.main.getVersion()+ ',' +
- (aws.source != null ? '1' : '0') + '\n');
+ w.write(aws.getGroupId() + ',' + aws.getArtifactId() + ',' + aws.getVersion() + ','
+ + (aws.getSourcesExists() == PRESENT ? '1' : '0') + '\n');
} catch (IOException e) {
throw new RuntimeException(e);
}
}
-
- private static ArtifactWithSource extract(ArtifactInfoGroup group) {
-
- ArtifactWithSource aws = new ArtifactWithSource();
-
- aws.main = group.getArtifactInfos()
- .stream()
- .filter( a -> a.getClassifier() == null)
- .findFirst()
- .orElse(null);
-
- if ( aws.main == null )
- return null;
-
- // TODO - need constant for 'sources'
- aws.source = group.getArtifactInfos().stream()
- .filter( a -> a.getVersion().equals(aws.main.getVersion()) && "sources".equals(a.getClassifier()) )
- .findFirst()
- .orElse(null);
-
- return aws;
- }
-
- static class ArtifactWithSource {
-
- private ArtifactInfo main;
-
- private ArtifactInfo source;
- }
}
diff --git a/maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/Main.java b/maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/Main.java
index 2165203..c588ba5 100644
--- a/maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/Main.java
+++ b/maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/Main.java
@@ -18,47 +18,6 @@
*/
package org.apache.sling.tooling.msra.impl;
-import java.io.BufferedOutputStream;
-import java.io.File;
-import java.io.FileOutputStream;
-import java.io.IOException;
-import java.io.OutputStreamWriter;
-import java.util.ArrayList;
-import java.util.Date;
-import java.util.List;
-import java.util.Objects;
-
-import org.apache.lucene.search.BooleanClause.Occur;
-import org.apache.lucene.search.BooleanQuery;
-import org.apache.lucene.search.Query;
-import org.apache.maven.index.ArtifactInfo;
-import org.apache.maven.index.ArtifactInfoGroup;
-import org.apache.maven.index.GroupedSearchRequest;
-import org.apache.maven.index.GroupedSearchResponse;
-import org.apache.maven.index.Indexer;
-import org.apache.maven.index.MAVEN;
-import org.apache.maven.index.context.IndexCreator;
-import org.apache.maven.index.context.IndexingContext;
-import org.apache.maven.index.expr.SourcedSearchExpression;
-import org.apache.maven.index.search.grouping.GAGrouping;
-import org.apache.maven.index.updater.IndexUpdateRequest;
-import org.apache.maven.index.updater.IndexUpdateResult;
-import org.apache.maven.index.updater.IndexUpdater;
-import org.apache.maven.index.updater.ResourceFetcher;
-import org.apache.maven.index.updater.WagonHelper;
-import org.apache.maven.wagon.Wagon;
-import org.apache.maven.wagon.events.TransferEvent;
-import org.apache.maven.wagon.events.TransferListener;
-import org.apache.maven.wagon.observers.AbstractTransferListener;
-import org.codehaus.plexus.DefaultContainerConfiguration;
-import org.codehaus.plexus.DefaultPlexusContainer;
-import org.codehaus.plexus.PlexusConstants;
-import org.codehaus.plexus.PlexusContainer;
-import org.codehaus.plexus.PlexusContainerException;
-import org.codehaus.plexus.component.repository.exception.ComponentLookupException;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
public class Main {
public static void main(String[] args) throws Exception {
@@ -67,171 +26,8 @@ public class Main {
throw new IllegalArgumentException("Please specify groupId to look for, e.g. org.apache*");
}
- Main main = new Main(args[0]);
- main.loadIndex();
- main.querySlingApiArtifacts();
- }
-
- private static final File OUT_DIR = new File("out");
-
- private final PlexusContainer plexusContainer;
-
- private final Indexer indexer;
-
- private final IndexUpdater indexUpdater;
-
- private final Wagon httpWagon;
-
- private IndexingContext centralContext;
-
- private final Logger logger = LoggerFactory.getLogger(getClass());
-
- private final String queryGroupId;
-
- public Main(String queryGroupId) throws PlexusContainerException, ComponentLookupException {
- // here we create Plexus container, the Maven default IoC container
- // Plexus falls outside of MI scope, just accept the fact that
- // MI is a Plexus component ;)
- // If needed more info, ask on Maven Users list or Plexus Users list
- // google is your friend!
- final DefaultContainerConfiguration config = new DefaultContainerConfiguration();
- config.setClassPathScanning(PlexusConstants.SCANNING_INDEX);
- this.plexusContainer = new DefaultPlexusContainer(config);
-
- // lookup the indexer components from plexus
- this.indexer = plexusContainer.lookup(Indexer.class);
- this.indexUpdater = plexusContainer.lookup(IndexUpdater.class);
- // lookup wagon used to remotely fetch index
- this.httpWagon = plexusContainer.lookup(Wagon.class, "https");
- this.queryGroupId = queryGroupId;
-
- }
-
- public void loadIndex() throws ComponentLookupException, IOException {
- // Files where local cache is (if any) and Lucene Index should be located
- File centralLocalCache = new File(OUT_DIR, "central-cache");
- File centralIndexDir = new File(OUT_DIR, "central-index");
-
- // Creators we want to use (search for fields it defines)
- List<IndexCreator> indexers = new ArrayList<>();
- indexers.add(plexusContainer.lookup(IndexCreator.class, "min"));
- indexers.add(plexusContainer.lookup(IndexCreator.class, "jarContent"));
- indexers.add(plexusContainer.lookup(IndexCreator.class, "maven-plugin"));
-
- // Create context for central repository index
- centralContext = indexer.createIndexingContext("central-context", "central", centralLocalCache, centralIndexDir,
- "https://repo1.maven.org/maven2", null, true, true, indexers);
-
- // Update the index (incremental update will happen if this is not 1st run and
- // files are not deleted)
- // This whole block below should not be executed on every app start, but rather
- // controlled by some configuration
- // since this block will always emit at least one HTTP GET. Central indexes are
- // updated once a week, but
- // other index sources might have different index publishing frequency.
- // Preferred frequency is once a week.
- logger.info("Updating Index. This might take a while on first run, so please be patient.");
- // Create ResourceFetcher implementation to be used with IndexUpdateRequest
- // Here, we use Wagon based one as shorthand, but all we need is a
- // ResourceFetcher implementation
- TransferListener listener = new AbstractTransferListener() {
- @Override
- public void transferStarted(TransferEvent transferEvent) {
- logger.info("Downloading {}", transferEvent.getResource().getName());
- }
-
- @Override
- public void transferCompleted(TransferEvent transferEvent) {
- logger.info("Done downloading {}", transferEvent.getResource().getName());
- }
- };
- ResourceFetcher resourceFetcher = new WagonHelper.WagonFetcher(httpWagon, listener, null, null);
-
- Date centralContextCurrentTimestamp = centralContext.getTimestamp();
- IndexUpdateRequest updateRequest = new IndexUpdateRequest(centralContext, resourceFetcher);
- IndexUpdateResult updateResult = indexUpdater.fetchAndUpdateIndex(updateRequest);
- if (updateResult.isFullUpdate()) {
- logger.info("Full update happened!");
- } else if (updateResult.getTimestamp().equals(centralContextCurrentTimestamp)) {
- logger.info("No update needed, index is up to date!");
- } else {
- logger.info("Incremental update happened, change covered {} - {} period.",
- centralContextCurrentTimestamp , updateResult.getTimestamp());
- }
- }
-
- public void querySlingApiArtifacts() throws IOException {
-
- Query groupId = indexer.constructQuery(MAVEN.GROUP_ID, new SourcedSearchExpression(queryGroupId));
- Query packaging = indexer.constructQuery(MAVEN.PACKAGING, new SourcedSearchExpression("pom"));
-
- BooleanQuery bq = new BooleanQuery.Builder()
- .add(groupId, Occur.MUST)
- .add(packaging, Occur.MUST_NOT)
- .build();
-
- searchAndDump( indexer, "all " + queryGroupId + " artifacts", bq );
- }
-
- public void searchAndDump(Indexer nexusIndexer, String descr, Query q) throws IOException {
- logger.info("Searching for {}", descr);
-
- GroupedSearchResponse response = nexusIndexer
- .searchGrouped(new GroupedSearchRequest(q, new GAGrouping(), centralContext));
-
- try ( FileOutputStream fos = new FileOutputStream(new File(OUT_DIR, "results.csv"));
- BufferedOutputStream bos = new BufferedOutputStream(fos);
- OutputStreamWriter w = new OutputStreamWriter(bos)) {
-
- response.getResults().values()
- .stream()
- .map( Main::extract )
- .filter( Objects::nonNull )
- .forEach( aws -> writeLine(w, aws));
-
- logger.info("Total hits: {}", response.getTotalHitsCount());
- }
-
- logger.info("Data written under {}", OUT_DIR);
- }
-
- private void writeLine(OutputStreamWriter w, ArtifactWithSource aws) {
- try {
- w.write(aws.main.getGroupId() + ',' +
- aws.main.getArtifactId()+ ',' +
- aws.main.getVersion()+ ',' +
- (aws.source != null ? '1' : '0') + '\n');
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- }
-
- private static ArtifactWithSource extract(ArtifactInfoGroup group) {
-
- ArtifactWithSource aws = new ArtifactWithSource();
-
- aws.main = group.getArtifactInfos()
- .stream()
- .filter( a -> a.getClassifier() == null)
- .findFirst()
- .orElse(null);
-
- if ( aws.main == null )
- return null;
-
- // TODO - need constant for 'sources'
- aws.source = group.getArtifactInfos().stream()
- .filter( a -> a.getVersion().equals(aws.main.getVersion()) && "sources".equals(a.getClassifier()) )
- .findFirst()
- .orElse(null);
-
- return aws;
- }
-
- static class ArtifactWithSource {
-
- private ArtifactInfo main;
-
- private ArtifactInfo source;
+ IndexQuerier q = new IndexQuerier();
+ q.loadIndex();
+ q.querySourceArtifacts(args[0]);
}
}
[sling-whiteboard] 02/09: Fix build and ignore out directory
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rombert pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/sling-whiteboard.git
commit 49c79ae90a797063408433a7e709c829a7be78b5
Author: Robert Munteanu <ro...@apache.org>
AuthorDate: Mon May 6 16:30:39 2019 +0200
Fix build and ignore out directory
---
maven-sources-report-aggregator/.gitignore | 1 +
maven-sources-report-aggregator/pom.xml | 30 +++++++++++++++++++++-
.../org/apache/sling/tooling/msra/impl/Main.java | 6 ++++-
3 files changed, 35 insertions(+), 2 deletions(-)
diff --git a/maven-sources-report-aggregator/.gitignore b/maven-sources-report-aggregator/.gitignore
new file mode 100644
index 0000000..6a3417b
--- /dev/null
+++ b/maven-sources-report-aggregator/.gitignore
@@ -0,0 +1 @@
+/out/
diff --git a/maven-sources-report-aggregator/pom.xml b/maven-sources-report-aggregator/pom.xml
index 3399ccd..1b91e4a 100644
--- a/maven-sources-report-aggregator/pom.xml
+++ b/maven-sources-report-aggregator/pom.xml
@@ -1,3 +1,21 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
@@ -31,7 +49,7 @@
<artifactId>slf4j-simple</artifactId>
<scope>compile</scope>
</dependency>
-
+
<!-- ensure javax.inject ends up in the fat jar -->
<dependency>
<groupId>javax.inject</groupId>
@@ -82,6 +100,16 @@
</execution>
</executions>
</plugin>
+ <!-- exclude local state files. not stored in target since they're expensive to regenerate -->
+ <plugin>
+ <groupId>org.apache.rat</groupId>
+ <artifactId>apache-rat-plugin</artifactId>
+ <configuration>
+ <excludes>
+ <exclude>out/**</exclude>
+ </excludes>
+ </configuration>
+ </plugin>
</plugins>
</build>
</project>
\ No newline at end of file
diff --git a/maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/Main.java b/maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/Main.java
index ce9d4a4..2165203 100644
--- a/maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/Main.java
+++ b/maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/Main.java
@@ -63,7 +63,11 @@ public class Main {
public static void main(String[] args) throws Exception {
- Main main = new Main("org.apache.sling*");
+ if ( args.length == 0 ) {
+ throw new IllegalArgumentException("Please specify groupId to look for, e.g. org.apache*");
+ }
+
+ Main main = new Main(args[0]);
main.loadIndex();
main.querySlingApiArtifacts();
}
[sling-whiteboard] 05/09: Updated README
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rombert pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/sling-whiteboard.git
commit 64fbcfee0b6d464345f6fef91f300ddf68f86426
Author: Robert Munteanu <ro...@apache.org>
AuthorDate: Mon May 6 16:57:13 2019 +0200
Updated README
---
maven-sources-report-aggregator/README.md | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/maven-sources-report-aggregator/README.md b/maven-sources-report-aggregator/README.md
new file mode 100644
index 0000000..5744393
--- /dev/null
+++ b/maven-sources-report-aggregator/README.md
@@ -0,0 +1,17 @@
+# Apache Sling Maven Repository Source Availability Reporter
+
+This project offers a CLI interface to checking the availability of source artifacts on Maven Central.
+
+## Usage
+
+```
+$ mvn clean verify
+$ java -jar target/org.apache.sling.tooling.maven-sources-report-aggregator-*-jar-with-dependencies.jar org.apache*
+```
+
+The runnable jar takes a single argument, which is the groupId to match artifacts against. It is valid to add a trailing star,
+which signals a prefix match.
+
+The initial run may be slow, as the project downloads and imports the Maven Central index using the [Maven Indexer](https://maven.apache.org/maven-indexer/indexer-reader/index.html).
+
+After running, the report is placed in `out/results.csv`.
\ No newline at end of file
[sling-whiteboard] 07/09: Formatting
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rombert pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/sling-whiteboard.git
commit 27d8acc163c43492c7b6c4d9f1649cdb02bc822e
Author: Robert Munteanu <ro...@apache.org>
AuthorDate: Tue May 7 11:43:47 2019 +0200
Formatting
---
.../main/java/org/apache/sling/tooling/msra/impl/IndexQuerier.java | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/IndexQuerier.java b/maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/IndexQuerier.java
index 5fa7353..5324732 100644
--- a/maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/IndexQuerier.java
+++ b/maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/IndexQuerier.java
@@ -157,12 +157,15 @@ public class IndexQuerier {
Query groupId = indexer.constructQuery(MAVEN.GROUP_ID, new SourcedSearchExpression(queryGroupId));
Query packaging = indexer.constructQuery(MAVEN.PACKAGING, new SourcedSearchExpression("pom"));
- BooleanQuery bq = new BooleanQuery.Builder().add(groupId, Occur.MUST).add(packaging, Occur.MUST_NOT).build();
+ BooleanQuery bq = new BooleanQuery.Builder()
+ .add(groupId, Occur.MUST)
+ .add(packaging, Occur.MUST_NOT)
+ .build();
searchAndDump(indexer, "all " + queryGroupId + " artifacts", bq);
}
- public void searchAndDump(Indexer nexusIndexer, String descr, Query q) throws IOException {
+ private void searchAndDump(Indexer nexusIndexer, String descr, Query q) throws IOException {
logger.info("Searching for {}", descr);
GroupedSearchResponse response = nexusIndexer
[sling-whiteboard] 08/09: Adjust artifactId and name
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rombert pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/sling-whiteboard.git
commit e7469e3afd92267fac08e46c576ca1c9f6f69c03
Author: Robert Munteanu <ro...@apache.org>
AuthorDate: Wed May 8 18:13:55 2019 +0200
Adjust artifactId and name
---
maven-sources-report-aggregator/README.md | 5 ++---
maven-sources-report-aggregator/pom.xml | 4 ++--
2 files changed, 4 insertions(+), 5 deletions(-)
diff --git a/maven-sources-report-aggregator/README.md b/maven-sources-report-aggregator/README.md
index 5744393..c385fab 100644
--- a/maven-sources-report-aggregator/README.md
+++ b/maven-sources-report-aggregator/README.md
@@ -1,4 +1,4 @@
-# Apache Sling Maven Repository Source Availability Reporter
+# Apache Sling Maven Central Source Availability Reporter
This project offers a CLI interface to checking the availability of source artifacts on Maven Central.
@@ -9,8 +9,7 @@ $ mvn clean verify
$ java -jar target/org.apache.sling.tooling.maven-sources-report-aggregator-*-jar-with-dependencies.jar org.apache*
```
-The runnable jar takes a single argument, which is the groupId to match artifacts against. It is valid to add a trailing star,
-which signals a prefix match.
+The runnable jar takes a single argument, which is the groupId to match artifacts against. It is valid to add a trailing star, which signals a prefix match.
The initial run may be slow, as the project downloads and imports the Maven Central index using the [Maven Indexer](https://maven.apache.org/maven-indexer/indexer-reader/index.html).
diff --git a/maven-sources-report-aggregator/pom.xml b/maven-sources-report-aggregator/pom.xml
index a3ebad4..4467545 100644
--- a/maven-sources-report-aggregator/pom.xml
+++ b/maven-sources-report-aggregator/pom.xml
@@ -27,9 +27,9 @@
<relativePath />
</parent>
- <artifactId>org.apache.sling.tooling.maven-sources-report-aggregator</artifactId>
+ <artifactId>org.apache.sling.tooling.maven-central-source-reporter</artifactId>
<version>0.0.1-SNAPSHOT</version>
- <name>Apache Sling - Maven Source Artifacts Report Aggregator</name>
+ <name>Apache Sling - Maven Central Source Reporter</name>
<dependencies>
<!-- Main dependency -->
[sling-whiteboard] 09/09: Renamed folder
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rombert pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/sling-whiteboard.git
commit 51f552e4024bdeb8f23178654d03fa94274b7c0b
Author: Robert Munteanu <ro...@apache.org>
AuthorDate: Wed May 8 18:14:47 2019 +0200
Renamed folder
---
.../.gitignore | 0
.../README.md | 0
.../pom.xml | 0
.../src/main/java/org/apache/sling/tooling/msra/impl/IndexQuerier.java | 0
.../src/main/java/org/apache/sling/tooling/msra/impl/Main.java | 0
5 files changed, 0 insertions(+), 0 deletions(-)
diff --git a/maven-sources-report-aggregator/.gitignore b/maven-central-source-reporter/.gitignore
similarity index 100%
rename from maven-sources-report-aggregator/.gitignore
rename to maven-central-source-reporter/.gitignore
diff --git a/maven-sources-report-aggregator/README.md b/maven-central-source-reporter/README.md
similarity index 100%
rename from maven-sources-report-aggregator/README.md
rename to maven-central-source-reporter/README.md
diff --git a/maven-sources-report-aggregator/pom.xml b/maven-central-source-reporter/pom.xml
similarity index 100%
rename from maven-sources-report-aggregator/pom.xml
rename to maven-central-source-reporter/pom.xml
diff --git a/maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/IndexQuerier.java b/maven-central-source-reporter/src/main/java/org/apache/sling/tooling/msra/impl/IndexQuerier.java
similarity index 100%
rename from maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/IndexQuerier.java
rename to maven-central-source-reporter/src/main/java/org/apache/sling/tooling/msra/impl/IndexQuerier.java
diff --git a/maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/Main.java b/maven-central-source-reporter/src/main/java/org/apache/sling/tooling/msra/impl/Main.java
similarity index 100%
rename from maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/Main.java
rename to maven-central-source-reporter/src/main/java/org/apache/sling/tooling/msra/impl/Main.java
[sling-whiteboard] 06/09: Remove unneeded indexers.
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rombert pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/sling-whiteboard.git
commit 08c6db208f8d813259187d4e03b74df32885a981
Author: Robert Munteanu <ro...@apache.org>
AuthorDate: Mon May 6 16:57:34 2019 +0200
Remove unneeded indexers.
---
.../src/main/java/org/apache/sling/tooling/msra/impl/IndexQuerier.java | 2 --
1 file changed, 2 deletions(-)
diff --git a/maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/IndexQuerier.java b/maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/IndexQuerier.java
index d5245c0..5fa7353 100644
--- a/maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/IndexQuerier.java
+++ b/maven-sources-report-aggregator/src/main/java/org/apache/sling/tooling/msra/impl/IndexQuerier.java
@@ -109,8 +109,6 @@ public class IndexQuerier {
// Creators we want to use (search for fields it defines)
List<IndexCreator> indexers = new ArrayList<>();
indexers.add(plexusContainer.lookup(IndexCreator.class, "min"));
- indexers.add(plexusContainer.lookup(IndexCreator.class, "jarContent"));
- indexers.add(plexusContainer.lookup(IndexCreator.class, "maven-plugin"));
// Create context for central repository index
centralContext = indexer.createIndexingContext("central-context", "central", centralLocalCache, centralIndexDir,