You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by GitBox <gi...@apache.org> on 2020/06/18 21:32:35 UTC

[GitHub] [camel] juanksega opened a new pull request #3934: camel-oaipmh: basic implementation

juanksega opened a new pull request #3934:
URL: https://github.com/apache/camel/pull/3934


   A first draft of the camel-oaipmh component. 
   
   Part of this code was imported from: [https://github.com/cbadenes/camel-oaipmh](camel-oaipmh)
   
   Important changes: 
   - Remove oai_dc specific classes.
   - Add https configuration.
   - Test component on a remote server.
   - Basic structure of a Producer.
   
   
   [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
   [ ] Each commit in the pull request should have a meaningful subject line and body.
   [ ] If you're unsure, you can format the pull request title like `[CAMEL-XXX] Fixes bug in camel-file component`, where you replace `CAMEL-XXX` with the appropriate JIRA issue.
   [ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
   [ ] Run `mvn clean install -Psourcecheck` in your module with source check enabled to make sure basic checks pass and there are no checkstyle violations. A more thorough check will be performed on your pull request automatically.
   Below are the contribution guidelines:
   https://github.com/apache/camel/blob/master/CONTRIBUTING.md


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel] omarsmak commented on a change in pull request #3934: [CAMEL-15253] camel-oaipmh - basic implementation

Posted by GitBox <gi...@apache.org>.
omarsmak commented on a change in pull request #3934:
URL: https://github.com/apache/camel/pull/3934#discussion_r463495764



##########
File path: components/camel-oaipmh/src/main/java/org/apache/camel/oaipmh/component/OAIPMHProducer.java
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.oaipmh.component;
+
+import java.net.URI;
+import java.util.List;
+
+import org.apache.camel.Exchange;
+import org.apache.camel.Message;
+import org.apache.camel.oaipmh.handler.Harvester;
+import org.apache.camel.oaipmh.handler.ProducerResponseHandler;
+import org.apache.camel.support.DefaultProducer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The OAIPMH producer.
+ */
+public class OAIPMHProducer extends DefaultProducer {
+
+    private static final Logger LOG = LoggerFactory.getLogger(OAIPMHProducer.class);
+    private OAIPMHEndpoint endpoint;
+
+    public OAIPMHProducer(OAIPMHEndpoint endpoint) {
+        super(endpoint);
+        this.endpoint = endpoint;
+    }
+
+    @Override
+    public void process(Exchange exchange) throws Exception {
+        Harvester harvester = new Harvester(
+                new ProducerResponseHandler(),
+                endpoint.getUrl(),
+                endpoint.getVerb(),
+                endpoint.getMetadataPrefix(),
+                endpoint.getUntil(),
+                endpoint.getFrom(),
+                endpoint.getSet(),
+                endpoint.getIdentitier());
+        overrideHarvesterConfigs(exchange.getIn(), harvester);
+        if (endpoint.isIgnoreSSLWarnings()) {
+            harvester.getHttpClient().setIgnoreSSLWarnings(true);
+        }
+        List<String> synHarvest = harvester.synHarvest(endpoint.isOnlyFirst());
+        exchange.getMessage().setBody(synHarvest);
+        if (endpoint.isOnlyFirst() && harvester.getResumptionToken() != null) {
+            exchange.getMessage().setHeader("CamelOaimphResumptionToken", harvester.getResumptionToken());
+        } else {
+            exchange.getMessage().removeHeader("CamelOaimphResumptionToken");
+        }
+    }
+
+    private void overrideHarvesterConfigs(Message msg, Harvester harvester) {
+        String header = msg.getHeader("CamelOaimphUrl", String.class);
+        if (header != null) {
+            harvester.setBaseURI(URI.create(header));
+        }
+
+        header = msg.getHeader("CamelOaimphEndpointUrl", String.class);
+        if (header != null) {

Review comment:
       I think these kind of validations, could be simplified into a function that takes lambda as argument, example:
   ```
   private <T> void checkAndSetConfigs(final Message message, final String key, final Consumer<T> fn, final Class<T> type) {
           final T header = message.getHeader(key, type);
           if (!ObjectHelper.isEmpty(header)) {
               fn.accept(header);
           }
       }
   ``` 
   to call it:
   ```
   checkAndSetConfigs(msg, "CamelOaimphVerb", harvester::setVerb, String.class);
   ```
   For the URI, you may need to add the a wrapper from URI to String somewhere that creates `URI.create(header)`

##########
File path: components/camel-oaipmh/src/main/java/org/apache/camel/oaipmh/handler/Harvester.java
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.oaipmh.handler;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.security.KeyManagementException;
+import java.security.KeyStoreException;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+import java.util.Optional;
+
+import javax.xml.parsers.ParserConfigurationException;
+
+import org.xml.sax.SAXException;
+
+import org.apache.camel.oaipmh.component.model.OAIPMHVerb;
+import org.apache.camel.oaipmh.model.OAIPMHResponse;
+import org.apache.camel.oaipmh.utils.OAIPMHHttpClient;
+
+public class Harvester {
+
+    private static final String NO_TOKEN = null;
+    private String resumptionToken = NO_TOKEN;
+    private URI baseURI;
+    private String verb;
+    private String metadata;
+    private String until;
+    private String from;
+    private String set;
+    private String identifier;
+    private OAIPMHHttpClient httpClient;
+    private ResponseHandler oaipmhResponseHandler;
+
+    private boolean empty;
+
+    public Harvester(ResponseHandler oaipmhResponseHandler, URI baseURI, String verb, String metadata, String until, String from, String set, String identifier) {
+        this.baseURI = baseURI;
+        this.verb = verb;
+        this.metadata = metadata;
+        this.until = until;
+        this.from = from;
+        this.set = set;
+        this.identifier = identifier;
+        this.httpClient = new OAIPMHHttpClient();
+        this.oaipmhResponseHandler = oaipmhResponseHandler;
+
+        if (OAIPMHVerb.valueOf(verb) == OAIPMHVerb.Identify) {
+            this.metadata = null;
+            this.until = null;
+            this.from = null;
+            this.set = null;
+            this.identifier = null;
+        }
+
+    }
+
+    private boolean harvest() throws IOException, URISyntaxException, ParserConfigurationException, SAXException, Exception {

Review comment:
       I don't think you need to throw all these exception since you are throwing `Exception`.

##########
File path: components/camel-oaipmh/pom.xml
##########
@@ -0,0 +1,140 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+
+    <modelVersion>4.0.0</modelVersion>
+
+    <parent>
+        <artifactId>components</artifactId>
+        <groupId>org.apache.camel</groupId>
+        <version>3.5.0-SNAPSHOT</version>
+    </parent>
+
+    <artifactId>camel-oaipmh</artifactId>
+    <packaging>jar</packaging>
+    <name>Camel :: OAIPMH</name>
+    <description>Camel OAI-PMH support</description>
+
+    <properties>
+    </properties>
+
+    <dependencyManagement>
+        <dependencies>
+            <!-- Camel BOM -->
+            <dependency>
+                <groupId>org.apache.camel</groupId>
+                <artifactId>camel-bom</artifactId>
+                <version>3.5.0-SNAPSHOT</version>
+                <scope>import</scope>
+                <type>pom</type>
+            </dependency>
+        </dependencies>
+    </dependencyManagement>
+
+    <dependencies>
+
+        <!-- camel -->
+        <dependency>
+            <groupId>org.apache.camel</groupId>
+            <artifactId>camel-support</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>joda-time</groupId>
+            <artifactId>joda-time</artifactId>
+            <version>${jodatime2-bundle-version}</version>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.httpcomponents</groupId>
+            <artifactId>httpclient</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>org.jsoup</groupId>
+            <artifactId>jsoup</artifactId>
+            <version>${jsoup-version}</version>
+        </dependency>
+
+        <!-- testing -->
+        <dependency>
+            <groupId>org.apache.camel</groupId>
+            <artifactId>camel-test</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.eclipse.jetty</groupId>
+            <artifactId>jetty-servlet</artifactId>
+            <version>${jetty-version}</version>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>commons-io</groupId>
+            <artifactId>commons-io</artifactId>
+            <scope>test</scope>
+        </dependency>
+    </dependencies>
+
+    <build>
+        <defaultGoal>install</defaultGoal>
+
+        <plugins>
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-compiler-plugin</artifactId>
+                <version>3.8.1</version>
+                <configuration>
+                    <source>1.8</source>
+                    <target>1.8</target>
+                </configuration>
+            </plugin>
+      
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-resources-plugin</artifactId>
+                <version>3.1.0</version>
+                <configuration>
+                    <encoding>UTF-8</encoding>
+                </configuration>
+            </plugin>
+
+            <!-- generate camel meta-data -->
+            <plugin>

Review comment:
       You don't need this plugin, is already included in the parent pom of the components.

##########
File path: components/camel-oaipmh/pom.xml
##########
@@ -0,0 +1,140 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+
+    <modelVersion>4.0.0</modelVersion>
+
+    <parent>
+        <artifactId>components</artifactId>
+        <groupId>org.apache.camel</groupId>
+        <version>3.5.0-SNAPSHOT</version>
+    </parent>
+
+    <artifactId>camel-oaipmh</artifactId>
+    <packaging>jar</packaging>
+    <name>Camel :: OAIPMH</name>
+    <description>Camel OAI-PMH support</description>
+
+    <properties>
+    </properties>
+
+    <dependencyManagement>
+        <dependencies>
+            <!-- Camel BOM -->
+            <dependency>
+                <groupId>org.apache.camel</groupId>
+                <artifactId>camel-bom</artifactId>
+                <version>3.5.0-SNAPSHOT</version>
+                <scope>import</scope>
+                <type>pom</type>
+            </dependency>
+        </dependencies>
+    </dependencyManagement>
+
+    <dependencies>
+
+        <!-- camel -->
+        <dependency>
+            <groupId>org.apache.camel</groupId>
+            <artifactId>camel-support</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>joda-time</groupId>
+            <artifactId>joda-time</artifactId>
+            <version>${jodatime2-bundle-version}</version>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.httpcomponents</groupId>
+            <artifactId>httpclient</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>org.jsoup</groupId>
+            <artifactId>jsoup</artifactId>
+            <version>${jsoup-version}</version>
+        </dependency>
+
+        <!-- testing -->
+        <dependency>
+            <groupId>org.apache.camel</groupId>
+            <artifactId>camel-test</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.eclipse.jetty</groupId>
+            <artifactId>jetty-servlet</artifactId>
+            <version>${jetty-version}</version>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>commons-io</groupId>
+            <artifactId>commons-io</artifactId>
+            <scope>test</scope>
+        </dependency>
+    </dependencies>
+
+    <build>
+        <defaultGoal>install</defaultGoal>
+
+        <plugins>

Review comment:
       and here

##########
File path: components/camel-oaipmh/src/main/java/org/apache/camel/oaipmh/handler/Harvester.java
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.oaipmh.handler;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.security.KeyManagementException;

Review comment:
       I see these imports are not used here

##########
File path: components/camel-oaipmh/src/main/java/org/apache/camel/oaipmh/utils/OAIPMHHttpClient.java
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.oaipmh.utils;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.charset.Charset;
+
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpResponse;
+import org.apache.http.RequestLine;
+import org.apache.http.client.ClientProtocolException;
+import org.apache.http.client.ResponseHandler;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.conn.ssl.SSLContextBuilder;
+import org.apache.http.conn.ssl.TrustSelfSignedStrategy;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.util.EntityUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class OAIPMHHttpClient {
+
+    private static final Logger LOG = LoggerFactory.getLogger(OAIPMHHttpClient.class);
+
+    private boolean ignoreSSLWarnings;
+
+    public String doRequest(URI baseURI, String verb, String set, String from, String until, String metadataPrefix, String token, String identifier) throws IOException, URISyntaxException, Exception {
+        CloseableHttpClient httpclient = getCloseableHttpClient();
+        try {
+
+            URIBuilder builder = new URIBuilder();
+            builder.setScheme(baseURI.getScheme())
+                    .setHost(baseURI.getHost())
+                    .setPort(baseURI.getPort())
+                    .setPath(baseURI.getPath())
+                    .addParameter("verb", verb);
+
+            if (baseURI.getQuery() != null && !baseURI.getQuery().isEmpty()) {
+                for (String param : baseURI.getQuery().split("&")) {
+                    builder.addParameter(param.split("=")[0], param.split("=")[1]);
+                }
+            }
+
+            if (identifier != null) {
+                builder.addParameter("identifier", identifier);
+            }
+
+            if (token != null) {
+                builder.addParameter("resumptionToken", token);
+            } else {
+                if (metadataPrefix != null) {
+                    builder.addParameter("metadataPrefix", metadataPrefix);
+                }
+                if (set != null) {
+                    builder.addParameter("set", set);
+                }
+                if (from != null) {
+                    builder.addParameter("from", from);
+                }
+                if (until != null) {
+                    builder.addParameter("until", until);
+                }
+            }
+
+            HttpGet httpget = new HttpGet(builder.build());
+
+            RequestLine requestLine = httpget.getRequestLine();
+
+            LOG.info("Executing request: {} ", requestLine);
+
+            // Create a custom response handler
+            ResponseHandler<String> responseHandler = new ResponseHandler<String>() {
+
+                @Override
+                public String handleResponse(final HttpResponse response) throws IOException {
+                    int status = response.getStatusLine().getStatusCode();
+                    if (status >= 200 && status < 300) {
+                        HttpEntity entity = response.getEntity();
+                        if (entity == null) {
+                            throw new IOException("No response received");
+                        }
+                        return EntityUtils.toString(entity, Charset.forName("UTF-8"));
+                    } else {
+                        throw new ClientProtocolException("Unexpected response status: " + status);
+                    }
+                }
+
+            };
+            String responseBody = httpclient.execute(httpget, responseHandler);
+
+//            String uri = requestLine.getUri();
+//            System.out.println(uri);
+//            String sha256Hex = DigestUtils.sha256Hex(uri.split(".org")[1]);
+//            System.out.println ("File:"+sha256Hex);
+//            BufferedWriter writer = new BufferedWriter(new FileWriter("/tmp/tests/test4/" + sha256Hex + ".xml"));
+//            writer.write(responseBody);
+//            writer.close();
+            LOG.debug("Response received: {}", responseBody);
+
+            return responseBody;
+        } finally {
+            httpclient.close();
+        }
+    }
+
+    protected CloseableHttpClient getCloseableHttpClient() throws Exception {
+        if (isIgnoreSSLWarnings()) {
+            try {
+                SSLContextBuilder builder = new SSLContextBuilder();
+                builder.loadTrustMaterial(null, new TrustSelfSignedStrategy());
+                SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(
+                        builder.build());
+                return HttpClients.custom().setSSLSocketFactory(
+                        sslsf).build();
+            } catch (Exception ex) {

Review comment:
       If you are catching the exception and then throw it again, makes no sense. However, I don't feel comfortable of throwing generic `Exception` here, please be specific on which exception you want to throw here

##########
File path: components/camel-oaipmh/src/main/java/org/apache/camel/oaipmh/utils/OAIPMHHttpClient.java
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.oaipmh.utils;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.charset.Charset;
+
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpResponse;
+import org.apache.http.RequestLine;
+import org.apache.http.client.ClientProtocolException;
+import org.apache.http.client.ResponseHandler;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.conn.ssl.SSLContextBuilder;
+import org.apache.http.conn.ssl.TrustSelfSignedStrategy;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.util.EntityUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class OAIPMHHttpClient {
+
+    private static final Logger LOG = LoggerFactory.getLogger(OAIPMHHttpClient.class);
+
+    private boolean ignoreSSLWarnings;
+
+    public String doRequest(URI baseURI, String verb, String set, String from, String until, String metadataPrefix, String token, String identifier) throws IOException, URISyntaxException, Exception {
+        CloseableHttpClient httpclient = getCloseableHttpClient();
+        try {
+
+            URIBuilder builder = new URIBuilder();
+            builder.setScheme(baseURI.getScheme())
+                    .setHost(baseURI.getHost())
+                    .setPort(baseURI.getPort())
+                    .setPath(baseURI.getPath())
+                    .addParameter("verb", verb);
+
+            if (baseURI.getQuery() != null && !baseURI.getQuery().isEmpty()) {
+                for (String param : baseURI.getQuery().split("&")) {
+                    builder.addParameter(param.split("=")[0], param.split("=")[1]);
+                }
+            }
+
+            if (identifier != null) {
+                builder.addParameter("identifier", identifier);
+            }
+
+            if (token != null) {
+                builder.addParameter("resumptionToken", token);
+            } else {
+                if (metadataPrefix != null) {
+                    builder.addParameter("metadataPrefix", metadataPrefix);
+                }
+                if (set != null) {
+                    builder.addParameter("set", set);
+                }
+                if (from != null) {
+                    builder.addParameter("from", from);
+                }
+                if (until != null) {
+                    builder.addParameter("until", until);
+                }
+            }
+
+            HttpGet httpget = new HttpGet(builder.build());
+
+            RequestLine requestLine = httpget.getRequestLine();
+
+            LOG.info("Executing request: {} ", requestLine);
+
+            // Create a custom response handler
+            ResponseHandler<String> responseHandler = new ResponseHandler<String>() {
+
+                @Override
+                public String handleResponse(final HttpResponse response) throws IOException {
+                    int status = response.getStatusLine().getStatusCode();
+                    if (status >= 200 && status < 300) {
+                        HttpEntity entity = response.getEntity();
+                        if (entity == null) {
+                            throw new IOException("No response received");
+                        }
+                        return EntityUtils.toString(entity, Charset.forName("UTF-8"));
+                    } else {
+                        throw new ClientProtocolException("Unexpected response status: " + status);
+                    }
+                }
+
+            };
+            String responseBody = httpclient.execute(httpget, responseHandler);
+
+//            String uri = requestLine.getUri();
+//            System.out.println(uri);
+//            String sha256Hex = DigestUtils.sha256Hex(uri.split(".org")[1]);
+//            System.out.println ("File:"+sha256Hex);
+//            BufferedWriter writer = new BufferedWriter(new FileWriter("/tmp/tests/test4/" + sha256Hex + ".xml"));
+//            writer.write(responseBody);
+//            writer.close();
+            LOG.debug("Response received: {}", responseBody);
+
+            return responseBody;
+        } finally {
+            httpclient.close();
+        }
+    }
+
+    protected CloseableHttpClient getCloseableHttpClient() throws Exception {
+        if (isIgnoreSSLWarnings()) {
+            try {
+                SSLContextBuilder builder = new SSLContextBuilder();

Review comment:
       This looks deprecated. 

##########
File path: components/camel-oaipmh/pom.xml
##########
@@ -0,0 +1,140 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+
+    <modelVersion>4.0.0</modelVersion>
+
+    <parent>
+        <artifactId>components</artifactId>
+        <groupId>org.apache.camel</groupId>
+        <version>3.5.0-SNAPSHOT</version>
+    </parent>
+
+    <artifactId>camel-oaipmh</artifactId>
+    <packaging>jar</packaging>
+    <name>Camel :: OAIPMH</name>
+    <description>Camel OAI-PMH support</description>
+
+    <properties>
+    </properties>
+
+    <dependencyManagement>

Review comment:
       You already having the parent dependency, I don't think you need to add the Camel BOM here

##########
File path: components/camel-oaipmh/src/main/java/org/apache/camel/oaipmh/component/OAIPMHProducer.java
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.oaipmh.component;
+
+import java.net.URI;
+import java.util.List;
+
+import org.apache.camel.Exchange;
+import org.apache.camel.Message;
+import org.apache.camel.oaipmh.handler.Harvester;
+import org.apache.camel.oaipmh.handler.ProducerResponseHandler;
+import org.apache.camel.support.DefaultProducer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The OAIPMH producer.
+ */
+public class OAIPMHProducer extends DefaultProducer {
+
+    private static final Logger LOG = LoggerFactory.getLogger(OAIPMHProducer.class);
+    private OAIPMHEndpoint endpoint;
+
+    public OAIPMHProducer(OAIPMHEndpoint endpoint) {
+        super(endpoint);
+        this.endpoint = endpoint;
+    }
+
+    @Override
+    public void process(Exchange exchange) throws Exception {
+        Harvester harvester = new Harvester(
+                new ProducerResponseHandler(),
+                endpoint.getUrl(),
+                endpoint.getVerb(),
+                endpoint.getMetadataPrefix(),
+                endpoint.getUntil(),
+                endpoint.getFrom(),
+                endpoint.getSet(),
+                endpoint.getIdentitier());
+        overrideHarvesterConfigs(exchange.getIn(), harvester);
+        if (endpoint.isIgnoreSSLWarnings()) {
+            harvester.getHttpClient().setIgnoreSSLWarnings(true);
+        }
+        List<String> synHarvest = harvester.synHarvest(endpoint.isOnlyFirst());
+        exchange.getMessage().setBody(synHarvest);
+        if (endpoint.isOnlyFirst() && harvester.getResumptionToken() != null) {
+            exchange.getMessage().setHeader("CamelOaimphResumptionToken", harvester.getResumptionToken());
+        } else {
+            exchange.getMessage().removeHeader("CamelOaimphResumptionToken");
+        }
+    }
+
+    private void overrideHarvesterConfigs(Message msg, Harvester harvester) {
+        String header = msg.getHeader("CamelOaimphUrl", String.class);

Review comment:
       I'd prefer to put all headers key string constants in a different class, [example](https://github.com/apache/camel/blob/master/components/camel-arangodb/src/main/java/org/apache/camel/component/arangodb/ArangoDbConstants.java#L19)

##########
File path: components/camel-oaipmh/src/main/java/org/apache/camel/oaipmh/utils/OAIPMHHttpClient.java
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.oaipmh.utils;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.charset.Charset;
+
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpResponse;
+import org.apache.http.RequestLine;
+import org.apache.http.client.ClientProtocolException;
+import org.apache.http.client.ResponseHandler;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.conn.ssl.SSLContextBuilder;
+import org.apache.http.conn.ssl.TrustSelfSignedStrategy;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.util.EntityUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class OAIPMHHttpClient {
+
+    private static final Logger LOG = LoggerFactory.getLogger(OAIPMHHttpClient.class);
+
+    private boolean ignoreSSLWarnings;
+
+    public String doRequest(URI baseURI, String verb, String set, String from, String until, String metadataPrefix, String token, String identifier) throws IOException, URISyntaxException, Exception {
+        CloseableHttpClient httpclient = getCloseableHttpClient();
+        try {

Review comment:
       You can replace it with:
   ```suggestion
          try (CloseableHttpClient httpclient = getCloseableHttpClient()) {
   ```

##########
File path: components/camel-oaipmh/pom.xml
##########
@@ -0,0 +1,140 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+
+    <modelVersion>4.0.0</modelVersion>
+
+    <parent>
+        <artifactId>components</artifactId>
+        <groupId>org.apache.camel</groupId>
+        <version>3.5.0-SNAPSHOT</version>
+    </parent>
+
+    <artifactId>camel-oaipmh</artifactId>
+    <packaging>jar</packaging>
+    <name>Camel :: OAIPMH</name>
+    <description>Camel OAI-PMH support</description>
+
+    <properties>
+    </properties>
+
+    <dependencyManagement>
+        <dependencies>
+            <!-- Camel BOM -->
+            <dependency>
+                <groupId>org.apache.camel</groupId>
+                <artifactId>camel-bom</artifactId>
+                <version>3.5.0-SNAPSHOT</version>
+                <scope>import</scope>
+                <type>pom</type>
+            </dependency>
+        </dependencies>
+    </dependencyManagement>
+
+    <dependencies>
+
+        <!-- camel -->
+        <dependency>
+            <groupId>org.apache.camel</groupId>
+            <artifactId>camel-support</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>joda-time</groupId>
+            <artifactId>joda-time</artifactId>
+            <version>${jodatime2-bundle-version}</version>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.httpcomponents</groupId>
+            <artifactId>httpclient</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>org.jsoup</groupId>
+            <artifactId>jsoup</artifactId>
+            <version>${jsoup-version}</version>
+        </dependency>
+
+        <!-- testing -->
+        <dependency>
+            <groupId>org.apache.camel</groupId>
+            <artifactId>camel-test</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.eclipse.jetty</groupId>
+            <artifactId>jetty-servlet</artifactId>
+            <version>${jetty-version}</version>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>commons-io</groupId>
+            <artifactId>commons-io</artifactId>
+            <scope>test</scope>
+        </dependency>
+    </dependencies>
+
+    <build>
+        <defaultGoal>install</defaultGoal>
+
+        <plugins>
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-compiler-plugin</artifactId>
+                <version>3.8.1</version>
+                <configuration>
+                    <source>1.8</source>
+                    <target>1.8</target>
+                </configuration>
+            </plugin>
+      
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-resources-plugin</artifactId>
+                <version>3.1.0</version>
+                <configuration>
+                    <encoding>UTF-8</encoding>
+                </configuration>
+            </plugin>
+
+            <!-- generate camel meta-data -->
+            <plugin>
+                <groupId>org.apache.camel</groupId>
+                <artifactId>camel-component-maven-plugin</artifactId>
+                <version>3.5.0-SNAPSHOT</version>
+                <executions>
+                    <execution>
+                        <id>generate</id>
+                        <goals>
+                            <goal>generate</goal>
+                        </goals>
+                        <phase>process-classes</phase>
+                    </execution>
+                </executions>
+            </plugin>
+            <plugin>
+                <groupId>org.codehaus.mojo</groupId>
+                <artifactId>build-helper-maven-plugin</artifactId>

Review comment:
       As well here

##########
File path: components/camel-oaipmh/src/main/java/org/apache/camel/oaipmh/utils/OAIPMHHttpClient.java
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.oaipmh.utils;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.charset.Charset;
+
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpResponse;
+import org.apache.http.RequestLine;
+import org.apache.http.client.ClientProtocolException;
+import org.apache.http.client.ResponseHandler;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.conn.ssl.SSLContextBuilder;
+import org.apache.http.conn.ssl.TrustSelfSignedStrategy;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.util.EntityUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class OAIPMHHttpClient {
+
+    private static final Logger LOG = LoggerFactory.getLogger(OAIPMHHttpClient.class);
+
+    private boolean ignoreSSLWarnings;
+
+    public String doRequest(URI baseURI, String verb, String set, String from, String until, String metadataPrefix, String token, String identifier) throws IOException, URISyntaxException, Exception {
+        CloseableHttpClient httpclient = getCloseableHttpClient();
+        try {
+
+            URIBuilder builder = new URIBuilder();
+            builder.setScheme(baseURI.getScheme())
+                    .setHost(baseURI.getHost())
+                    .setPort(baseURI.getPort())
+                    .setPath(baseURI.getPath())
+                    .addParameter("verb", verb);
+
+            if (baseURI.getQuery() != null && !baseURI.getQuery().isEmpty()) {
+                for (String param : baseURI.getQuery().split("&")) {
+                    builder.addParameter(param.split("=")[0], param.split("=")[1]);

Review comment:
       I am just wondering, do we need to check that we really have two params split with `=`? 

##########
File path: components/camel-oaipmh/src/main/java/org/apache/camel/oaipmh/utils/OAIPMHHttpClient.java
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.oaipmh.utils;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.charset.Charset;
+
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpResponse;
+import org.apache.http.RequestLine;
+import org.apache.http.client.ClientProtocolException;
+import org.apache.http.client.ResponseHandler;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.conn.ssl.SSLContextBuilder;
+import org.apache.http.conn.ssl.TrustSelfSignedStrategy;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.util.EntityUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class OAIPMHHttpClient {
+
+    private static final Logger LOG = LoggerFactory.getLogger(OAIPMHHttpClient.class);
+
+    private boolean ignoreSSLWarnings;
+
+    public String doRequest(URI baseURI, String verb, String set, String from, String until, String metadataPrefix, String token, String identifier) throws IOException, URISyntaxException, Exception {
+        CloseableHttpClient httpclient = getCloseableHttpClient();
+        try {
+
+            URIBuilder builder = new URIBuilder();
+            builder.setScheme(baseURI.getScheme())
+                    .setHost(baseURI.getHost())
+                    .setPort(baseURI.getPort())
+                    .setPath(baseURI.getPath())
+                    .addParameter("verb", verb);
+
+            if (baseURI.getQuery() != null && !baseURI.getQuery().isEmpty()) {
+                for (String param : baseURI.getQuery().split("&")) {
+                    builder.addParameter(param.split("=")[0], param.split("=")[1]);
+                }
+            }
+
+            if (identifier != null) {
+                builder.addParameter("identifier", identifier);
+            }
+
+            if (token != null) {
+                builder.addParameter("resumptionToken", token);
+            } else {
+                if (metadataPrefix != null) {
+                    builder.addParameter("metadataPrefix", metadataPrefix);
+                }
+                if (set != null) {
+                    builder.addParameter("set", set);
+                }
+                if (from != null) {
+                    builder.addParameter("from", from);
+                }
+                if (until != null) {
+                    builder.addParameter("until", until);
+                }
+            }
+
+            HttpGet httpget = new HttpGet(builder.build());
+
+            RequestLine requestLine = httpget.getRequestLine();
+
+            LOG.info("Executing request: {} ", requestLine);
+
+            // Create a custom response handler
+            ResponseHandler<String> responseHandler = new ResponseHandler<String>() {
+
+                @Override
+                public String handleResponse(final HttpResponse response) throws IOException {
+                    int status = response.getStatusLine().getStatusCode();
+                    if (status >= 200 && status < 300) {
+                        HttpEntity entity = response.getEntity();
+                        if (entity == null) {
+                            throw new IOException("No response received");
+                        }
+                        return EntityUtils.toString(entity, Charset.forName("UTF-8"));
+                    } else {
+                        throw new ClientProtocolException("Unexpected response status: " + status);
+                    }
+                }
+
+            };
+            String responseBody = httpclient.execute(httpget, responseHandler);
+
+//            String uri = requestLine.getUri();

Review comment:
       unnecessary comments here




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel] omarsmak merged pull request #3934: [CAMEL-15253] camel-oaipmh - basic implementation

Posted by GitBox <gi...@apache.org>.
omarsmak merged pull request #3934:
URL: https://github.com/apache/camel/pull/3934


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel] juanksega commented on pull request #3934: camel-oaipmh: basic implementation

Posted by GitBox <gi...@apache.org>.
juanksega commented on pull request #3934:
URL: https://github.com/apache/camel/pull/3934#issuecomment-649909993


   Hi @DenisIstomin @zregvart @oscerd 
   
   I pulled the last commits from the master branch of apache/camel, also I added a Producer implementation and some tests. Please let me know if you have suggestions.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel] omarsmak commented on pull request #3934: [CAMEL-15253] camel-oaipmh - basic implementation

Posted by GitBox <gi...@apache.org>.
omarsmak commented on pull request #3934:
URL: https://github.com/apache/camel/pull/3934#issuecomment-673385751


   @oscerd we shall merge this one?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel] oscerd commented on pull request #3934: [CAMEL-15253] camel-oaipmh - basic implementation

Posted by GitBox <gi...@apache.org>.
oscerd commented on pull request #3934:
URL: https://github.com/apache/camel/pull/3934#issuecomment-673388281


   Yeah, let's merge and revisit and improving later


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel] bedlaj commented on pull request #3934: camel-oaipmh: basic implementation

Posted by GitBox <gi...@apache.org>.
bedlaj commented on pull request #3934:
URL: https://github.com/apache/camel/pull/3934#issuecomment-650567476


   Good start. Since this is draft, i will give a few suggestions, instead of full review.
   
   - Please avoid using lombok there are many good reasons to not using that.
   - Use jsoup and joda-time versions provided in parent pom.
   - In final version, there must be all remote http calls removed, instead setup mock or use testcontainers.
   - Thread.sleep in tests can be replaced with proper assertions.
   - Never use `assert` keyword in unit tests.
   - Do not use `@author` annotations in javadoc.
   - Is safe to assume that response will be always UTF-8 and in Zulu timezone?
   - Headers should follow CamelOaimph* pattern, insead of oaimph.*
   - Endpoint class should contain some field annotated with `@UriPath`
   - For consumer use AsyncCallback in AsyncProcessor instead of try-catch-finally (Look eg to WsConsumer for inspiration)
   - Is `build-helper-maven-plugin` required? I dont see any custom generated (re)sources.
   
   And some obvious things, which I believe you know about.
   
   - Checkstyle violations
   - Missing licence headers
   - Logging to console instead of file
   - Missing Jira issue
   - Commented out UriEndpoint annotation (+category attribute)
   - Full build needed, to generate endpoint-dsl, component-dsl, ...
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [camel] juanksega commented on pull request #3934: [CAMEL-15253] camel-oaipmh - basic implementation

Posted by GitBox <gi...@apache.org>.
juanksega commented on pull request #3934:
URL: https://github.com/apache/camel/pull/3934#issuecomment-670323413


   Hi @bedlaj @omarsmak @DenisIstomin 
   
   Thanks so much for your feedback. I think I fixed all your suggestions. Please, let me know if you have other suggestions.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org