You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@stormcrawler.apache.org by jn...@apache.org on 2024/05/17 13:35:23 UTC
(incubator-stormcrawler) branch main updated: 1024 Playwright protocol implementation, fixes #1024 (#1228)
This is an automated email from the ASF dual-hosted git repository.
jnioche pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-stormcrawler.git
The following commit(s) were added to refs/heads/main by this push:
new 2eafa0a3 1024 Playwright protocol implementation, fixes #1024 (#1228)
2eafa0a3 is described below
commit 2eafa0a3bd1b63a55152db4e05885017b460c935
Author: Julien Nioche <ju...@digitalpebble.com>
AuthorDate: Fri May 17 14:35:18 2024 +0100
1024 Playwright protocol implementation, fixes #1024 (#1228)
* Playwright protocol implementation, fixes #1024
Signed-off-by: Julien Nioche <ju...@digitalpebble.com>
* Implements a unit test for Playwright PR (#1229)
* Implements a unit test to actually test that playwright is rendering HTML/JS content correctly.
Adds an option to specify wait until state to ensure JS is executed properly
Removes PDF test, which was a "false friend" because it was returned by the mock server with an HTML mimetype and the actual returned PDF object was corrupted.
* Skip playwright tests in CI as GH action cannot execute them due to missing libs
* Fix formatting
---------
Signed-off-by: Julien Nioche <ju...@digitalpebble.com>
Co-authored-by: Richard Zowalla <13...@users.noreply.github.com>
---
.github/workflows/maven.yml | 2 +-
THIRD-PARTY.txt | 5 +
external/playwright/README.md | 30 +++
external/playwright/playwright-conf.yaml | 10 +
external/playwright/pom.xml | 81 ++++++
.../protocol/playwright/HttpProtocol.java | 297 +++++++++++++++++++++
.../protocol/playwright/LocalResourceHandler.java | 66 +++++
.../protocol/playwright/ProtocolTest.java | 168 ++++++++++++
.../src/test/resources/dynamic-scraping.html | 32 +++
pom.xml | 1 +
10 files changed, 691 insertions(+), 1 deletion(-)
diff --git a/.github/workflows/maven.yml b/.github/workflows/maven.yml
index 486c321f..5e2f91b5 100644
--- a/.github/workflows/maven.yml
+++ b/.github/workflows/maven.yml
@@ -44,4 +44,4 @@ jobs:
distribution: adopt
java-version: ${{ matrix.java }}
- name: Build with Maven
- run: mvn -B --no-transfer-progress package --file pom.xml
+ run: mvn -B --no-transfer-progress package --file pom.xml -DCI_ENV=true
diff --git a/THIRD-PARTY.txt b/THIRD-PARTY.txt
index d7911348..e23eb21f 100644
--- a/THIRD-PARTY.txt
+++ b/THIRD-PARTY.txt
@@ -136,6 +136,7 @@ List of third-party dependencies grouped by their license type.
* FindBugs-jsr305 (com.google.code.findbugs:jsr305:3.0.2 - http://findbugs.sourceforge.net/)
* Google Android Annotations Library (com.google.android:annotations:4.1.1.4 - http://source.android.com/)
* Graphite Integration for Metrics (io.dropwizard.metrics:metrics-graphite:3.2.6 - http://metrics.dropwizard.io/metrics-graphite/)
+ * Gson (com.google.code.gson:gson:2.8.9 - https://github.com/google/gson/gson)
* Gson (com.google.code.gson:gson:2.9.0 - https://github.com/google/gson/gson)
* Guava: Google Core Libraries for Java (com.google.guava:guava:18.0 - http://code.google.com/p/guava-libraries/guava)
* Guava: Google Core Libraries for Java (com.google.guava:guava:31.1-android - https://github.com/google/guava)
@@ -234,6 +235,7 @@ List of third-party dependencies grouped by their license type.
* OpenTelemetry Java (io.opentelemetry:opentelemetry-sdk-trace:1.35.0 - https://github.com/open-telemetry/opentelemetry-java)
* OpenTelemetry Java (io.opentelemetry:opentelemetry-semconv:1.15.0-alpha - https://github.com/open-telemetry/opentelemetry-java)
* OpenTelemetry Semantic Conventions Java (io.opentelemetry.semconv:opentelemetry-semconv:1.23.1-alpha - https://github.com/open-telemetry/semantic-conventions-java)
+ * org.opentest4j:opentest4j (org.opentest4j:opentest4j:1.2.0 - https://github.com/ota4j-team/opentest4j)
* org.seleniumhq.selenium:selenium-api (org.seleniumhq.selenium:selenium-api:4.18.1 - https://selenium.dev/)
* org.seleniumhq.selenium:selenium-http (org.seleniumhq.selenium:selenium-http:4.18.1 - https://selenium.dev/)
* org.seleniumhq.selenium:selenium-json (org.seleniumhq.selenium:selenium-json:4.18.1 - https://selenium.dev/)
@@ -245,6 +247,9 @@ List of third-party dependencies grouped by their license type.
* parso (com.epam:parso:2.0.14 - https://github.com/epam/parso)
* PDFBox JBIG2 ImageIO plugin (org.apache.pdfbox:jbig2-imageio:3.0.4 - https://www.apache.org/jbig2-imageio/)
* perfmark:perfmark-api (io.perfmark:perfmark-api:0.25.0 - https://github.com/perfmark/perfmark)
+ * Playwright - Driver (com.microsoft.playwright:driver:1.41.2 - https://github.com/microsoft/playwright-java/driver)
+ * Playwright - Drivers For All Platforms (com.microsoft.playwright:driver-bundle:1.41.2 - https://github.com/microsoft/playwright-java/driver-bundle)
+ * Playwright - Main Library (com.microsoft.playwright:playwright:1.41.2 - https://github.com/microsoft/playwright-java/playwright)
* proto-google-common-protos (com.google.api.grpc:proto-google-common-protos:2.9.0 - https://github.com/googleapis/java-iam/proto-google-common-protos)
* rank-eval (org.opensearch.plugin:rank-eval-client:2.12.0 - https://github.com/opensearch-project/OpenSearch.git)
* rest (org.opensearch.client:opensearch-rest-client:2.12.0 - https://github.com/opensearch-project/OpenSearch.git)
diff --git a/external/playwright/README.md b/external/playwright/README.md
new file mode 100644
index 00000000..b0d6fdb4
--- /dev/null
+++ b/external/playwright/README.md
@@ -0,0 +1,30 @@
+# Playwright
+Protocol implementation for Apache StormCrawler (Incubating) based on Playwright
+
+## Standalone Chrome
+
+Playwright installs the browsers it needs locally, which can take more than 1 GB disk space and takes some time. Instead, you can install Chrome separately and get Playwright to connect to its CDP.
+
+[Chrome for Testing](https://github.com/GoogleChromeLabs/chrome-for-testing) is the recommended way of installing a standalone version of Chrome.
+
+You can start it like so
+
+```
+ ./chrome --remote-debugging-port=9222 --user-data-dir=remote-profile --headless &
+```
+NOTE this is how ChromeDriver launches Chrome
+
+```
+chrome --allow-pre-commit-input --disable-background-networking --disable-client-side-phishing-detection --disable-default-apps --disable-dev-shm-usage --disable-gpu --disable-hang-monitor --disable-popup-blocking --disable-prompt-on-repost --disable-software-rasterizer --disable-sync --dns-prefetch-disable --enable-automation --enable-logging --headless --log-level=0 --mute-audio --no-first-run --no-sandbox --no-service-autorun --password-store=basic --remote-debugging-port=9222 --use-m [...]
+```
+
+Then URL for the CDP connection is configured with _playwright.cdp.url_.
+
+Alternatively, you can still rely on what Playwright installs but do it manually and just for the browser type you want
+
+```
+mvn exec:java -e -Dexec.mainClass=com.microsoft.playwright.CLI -Dexec.args="install chromium"
+```
+
+The setting `playwright.skip.download` to `true` in the configuration will assume that the browser has been installed and will not trigger the installation of all the different browsers.
+
diff --git a/external/playwright/playwright-conf.yaml b/external/playwright/playwright-conf.yaml
new file mode 100644
index 00000000..4298ac2d
--- /dev/null
+++ b/external/playwright/playwright-conf.yaml
@@ -0,0 +1,10 @@
+config:
+ # standard SC configuration
+ http.agent: "StormCrawlerPlaywright"
+ http.timeout: 5000
+
+ # playwright.cdp.url: "http://localhost:9222"
+ playwright.skip.download: true
+ # com.microsoft.playwright.options.WaitUntilState
+ # playwright.load.event: "domcontentloaded"
+
diff --git a/external/playwright/pom.xml b/external/playwright/pom.xml
new file mode 100644
index 00000000..676ee2fb
--- /dev/null
+++ b/external/playwright/pom.xml
@@ -0,0 +1,81 @@
+<?xml version="1.0" encoding="UTF-8"?>
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.stormcrawler</groupId>
+ <artifactId>stormcrawler-external</artifactId>
+ <version>3.1.0-SNAPSHOT</version>
+ <relativePath>../pom.xml</relativePath>
+ </parent>
+
+ <artifactId>stormcrawler-playwright</artifactId>
+ <packaging>jar</packaging>
+
+ <name>stormcrawler-playwright</name>
+ <url>https://github.com/apache/incubator-stormcrawler/tree/master/external/playwright</url>
+ <description>Playwright-based protocol for StormCrawler</description>
+
+ <properties>
+ <playwright.version>1.43.0</playwright.version>
+ <awaitility.version>4.2.1</awaitility.version>
+ <wiremock.version>3.5.4</wiremock.version>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>com.microsoft.playwright</groupId>
+ <artifactId>playwright</artifactId>
+ <version>${playwright.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.stormcrawler</groupId>
+ <artifactId>stormcrawler-core</artifactId>
+ <version>${project.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.wiremock</groupId>
+ <artifactId>wiremock</artifactId>
+ <version>${wiremock.version}</version>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.awaitility</groupId>
+ <artifactId>awaitility</artifactId>
+ <version>${awaitility.version}</version>
+ <scope>test</scope>
+ </dependency>
+ </dependencies>
+
+</project>
diff --git a/external/playwright/src/main/java/org/apache/stormcrawler/protocol/playwright/HttpProtocol.java b/external/playwright/src/main/java/org/apache/stormcrawler/protocol/playwright/HttpProtocol.java
new file mode 100644
index 00000000..30d69ec0
--- /dev/null
+++ b/external/playwright/src/main/java/org/apache/stormcrawler/protocol/playwright/HttpProtocol.java
@@ -0,0 +1,297 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.stormcrawler.protocol.playwright;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.microsoft.playwright.Browser;
+import com.microsoft.playwright.Browser.NewContextOptions;
+import com.microsoft.playwright.BrowserContext;
+import com.microsoft.playwright.BrowserType;
+import com.microsoft.playwright.Page;
+import com.microsoft.playwright.Playwright;
+import com.microsoft.playwright.Playwright.CreateOptions;
+import com.microsoft.playwright.Tracing;
+import com.microsoft.playwright.options.Proxy;
+import com.microsoft.playwright.options.WaitUntilState;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.attribute.FileAttribute;
+import java.time.Instant;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.storm.Config;
+import org.apache.storm.utils.MutableInt;
+import org.apache.stormcrawler.Metadata;
+import org.apache.stormcrawler.persistence.Status;
+import org.apache.stormcrawler.protocol.AbstractHttpProtocol;
+import org.apache.stormcrawler.protocol.Protocol;
+import org.apache.stormcrawler.protocol.ProtocolResponse;
+import org.apache.stormcrawler.util.ConfUtils;
+import org.slf4j.LoggerFactory;
+
+public class HttpProtocol extends AbstractHttpProtocol {
+
+ private static final org.slf4j.Logger LOG = LoggerFactory.getLogger(HttpProtocol.class);
+
+ private Browser browser;
+
+ public static final String MD_KEY_START = "playwright.protocol.start";
+ public static final String MD_KEY_END = "playwright.protocol.end";
+ public static final String MD_TRACE = "playwright.trace";
+ public static final String MD_EVALUATIONS = "playwright.evaluations";
+ public static final String MD_SKIPS = "playwright.skip.resource.types";
+
+ private int timeout = 10000;
+
+ private BrowserContext context;
+
+ private Set<String> resourceTypesToSkip;
+
+ private ObjectMapper mapper;
+
+ // The Page.evaluate() API can run a JavaScript function in the context of the
+ // web page and bring results back to the Playwright environment. Browser
+ // globals like window and document can be used in evaluate.
+ private List<String> evaluations;
+
+ private WaitUntilState loadEvent;
+
+ @Override
+ public void configure(final Config conf) {
+ super.configure(conf);
+
+ // "http://localhost:9222"
+ final String CDP_URL = ConfUtils.getString(conf, "playwright.cdp.url");
+
+ boolean skipDownloads = ConfUtils.getBoolean(conf, "playwright.skip.download", false);
+
+ String loadEventString = ConfUtils.getString(conf, "playwright.load.event", "load");
+ // map load event string to playwright native wait stat
+ switch (loadEventString) {
+ case "domcontentloaded":
+ loadEvent = WaitUntilState.DOMCONTENTLOADED;
+ break;
+ case "networkidle":
+ loadEvent = WaitUntilState.NETWORKIDLE;
+ break;
+ default:
+ loadEvent = WaitUntilState.LOAD;
+ break;
+ }
+
+ final CreateOptions creationOptions = new CreateOptions();
+ final Map<String, String> env = new HashMap<>();
+
+ if (CDP_URL != null) {
+ skipDownloads = true;
+ }
+
+ // skip browser download if relying on an existing one
+ if (skipDownloads) {
+ env.put("PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD", "true");
+ }
+
+ creationOptions.setEnv(env);
+
+ BrowserType btype = Playwright.create(creationOptions).chromium();
+ if (CDP_URL != null) {
+ browser = btype.connectOverCDP(CDP_URL);
+ } else {
+ browser = btype.launch();
+ }
+
+ timeout = ConfUtils.getInt(conf, "http.timeout", timeout);
+
+ final String ua = getAgentString(conf);
+
+ NewContextOptions b_c_options =
+ new Browser.NewContextOptions().setIsMobile(false).setUserAgent(ua);
+
+ // global proxy
+ String proxyServer = ConfUtils.getString(conf, "http.proxy");
+ String proxyUser = ConfUtils.getString(conf, "http.proxy.username");
+ String proxyPwd = ConfUtils.getString(conf, "http.proxy.password");
+
+ final Proxy globalProxy = getProxy(proxyServer, proxyUser, proxyPwd);
+ if (globalProxy != null) {
+ b_c_options.setProxy(globalProxy);
+ b_c_options.setIgnoreHTTPSErrors(true);
+ }
+
+ context = browser.newContext(b_c_options);
+
+ context.setDefaultTimeout(timeout);
+
+ // list of resource types to skip
+ // document, stylesheet, image, media, font, script, texttrack, xhr, fetch,
+ // eventsource, websocket, manifest, other
+ resourceTypesToSkip = new java.util.HashSet<>();
+ for (String v : ConfUtils.loadListFromConf(MD_SKIPS, conf)) {
+ resourceTypesToSkip.add(v);
+ }
+
+ mapper = new ObjectMapper();
+
+ // expressions to evaluate
+ evaluations = ConfUtils.loadListFromConf(MD_EVALUATIONS, conf);
+ }
+
+ @Override
+ public ProtocolResponse getProtocolOutput(String url, Metadata md) throws Exception {
+
+ // https://github.com/microsoft/playwright-java#is-playwright-thread-safe
+ synchronized (this) {
+
+ // tracing// Start tracing before creating / navigating a page.
+ if (md.containsKey(MD_TRACE)) {
+ context.tracing()
+ .start(
+ new Tracing.StartOptions()
+ .setScreenshots(false)
+ .setSnapshots(true)
+ .setSources(true));
+ }
+
+ final Metadata responseMetaData = new Metadata();
+ responseMetaData.addValue(MD_KEY_START, Instant.now().toString());
+
+ final MutableInt status = new MutableInt(-1);
+ byte[] content = new byte[0];
+
+ try (Page page = context.newPage()) {
+
+ page.onResponse(
+ response -> {
+ // make sure that this applies to the main page
+ if (response.url().equals(url)) {
+ // redirection?
+ if (Status.REDIRECTION.equals(
+ Status.fromHTTPCode(response.status()))) {
+ status.set(response.status());
+ response.allHeaders()
+ .forEach(
+ (k, v) -> {
+ responseMetaData.addValue(k, v);
+ });
+ }
+ }
+ });
+
+ page.onPageError(
+ handler -> {
+ // this applies to any resource - not just the main page
+ LOG.debug("Error when loading {} {}", url, handler);
+ });
+
+ // NOTE: The handler will only be called for the first url if the
+ // response is a redirect.
+ page.route(
+ _url -> true,
+ route -> {
+ // abort if we know the main page is a redirection
+ if (status.get() != -1) {
+ LOG.debug("Aborting request for {}", route.request().url());
+ route.abort();
+ } else if (resourceTypesToSkip.contains(
+ route.request().resourceType())) {
+ route.abort();
+ } else {
+ route.resume();
+ }
+ });
+
+ // let playwright do the content loading
+ com.microsoft.playwright.Response response =
+ page.navigate(
+ url,
+ new Page.NavigateOptions()
+ .setTimeout(timeout)
+ .setWaitUntil(loadEvent));
+
+ // the status is not set unless
+ // a redirection
+ if (status.get() == -1) {
+ response.allHeaders()
+ .forEach(
+ (k, v) -> {
+ responseMetaData.addValue(k, v);
+ });
+
+ if (Status.FETCHED == Status.fromHTTPCode(response.status())) {
+ // retrieve the rendered content
+ content = page.content().getBytes(StandardCharsets.UTF_8);
+ }
+
+ status.set(response.status());
+
+ // evaluate an expression and store the results
+ // in the metadata using the same string as key
+ for (String expression : evaluations) {
+ Object performance = page.evaluate(expression);
+ if (performance != null) {
+ String json =
+ mapper.writerWithDefaultPrettyPrinter()
+ .writeValueAsString(performance);
+ responseMetaData.setValue(expression, json);
+ }
+ }
+ }
+ }
+
+ if (md.containsKey(MD_TRACE)) {
+ Path tmp = Files.createTempFile("trace-", ".zip", new FileAttribute[0]);
+ context.tracing().stop(new Tracing.StopOptions().setPath(tmp));
+ responseMetaData.setValue(MD_TRACE, tmp.toString());
+ }
+
+ responseMetaData.addValue(MD_KEY_END, Instant.now().toString());
+
+ return new ProtocolResponse(content, status.get(), responseMetaData);
+ }
+ }
+
+ /** Returns a proxy object if required * */
+ private Proxy getProxy(String proxyserver, String proxyuser, String proxypwd) {
+ if (proxyserver == null) return null;
+
+ Proxy proxy = new Proxy(proxyserver);
+ if (proxyuser != null) {
+ proxy.setUsername(proxyuser);
+ }
+ if (proxypwd != null) {
+ proxy.setPassword(proxypwd);
+ }
+ return proxy;
+ }
+
+ @Override
+ public void cleanup() {
+ synchronized (this) {
+ super.cleanup();
+ context.close();
+ browser.close();
+ }
+ }
+
+ public static void main(String[] args) throws Exception {
+ Protocol.main(new HttpProtocol(), args);
+ }
+}
diff --git a/external/playwright/src/test/java/org/apache/stormcrawler/protocol/playwright/LocalResourceHandler.java b/external/playwright/src/test/java/org/apache/stormcrawler/protocol/playwright/LocalResourceHandler.java
new file mode 100644
index 00000000..7c3cf8d9
--- /dev/null
+++ b/external/playwright/src/test/java/org/apache/stormcrawler/protocol/playwright/LocalResourceHandler.java
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.stormcrawler.protocol.playwright;
+
+import jakarta.servlet.http.HttpServletResponse;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import org.apache.commons.io.IOUtils;
+import org.eclipse.jetty.server.Request;
+import org.eclipse.jetty.server.handler.AbstractHandler;
+
+/** Serve the content of the resources if any found * */
+public class LocalResourceHandler extends AbstractHandler {
+
+ @Override
+ public void handle(
+ String s,
+ Request baseRequest,
+ jakarta.servlet.http.HttpServletRequest httpServletRequest,
+ jakarta.servlet.http.HttpServletResponse response)
+ throws IOException {
+ if (response.isCommitted() || baseRequest.isHandled()) return;
+
+ // does the URL have a file element?
+ if (s.length() < 2) return;
+
+ baseRequest.setHandled(true);
+
+ final String file = s.substring(1);
+ InputStream stream = getClass().getClassLoader().getResourceAsStream(file);
+
+ // not found?
+ if (stream == null) {
+ response.setStatus(HttpServletResponse.SC_NOT_FOUND);
+ return;
+ }
+
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ IOUtils.copy(stream, baos);
+ byte[] barray = baos.toByteArray();
+
+ response.setHeader("key", "value");
+ response.setStatus(HttpServletResponse.SC_OK);
+ response.setContentType("text/html");
+ try (OutputStream out = response.getOutputStream()) {
+ out.write(barray);
+ }
+ }
+}
diff --git a/external/playwright/src/test/java/org/apache/stormcrawler/protocol/playwright/ProtocolTest.java b/external/playwright/src/test/java/org/apache/stormcrawler/protocol/playwright/ProtocolTest.java
new file mode 100644
index 00000000..3e3eeb52
--- /dev/null
+++ b/external/playwright/src/test/java/org/apache/stormcrawler/protocol/playwright/ProtocolTest.java
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.stormcrawler.protocol.playwright;
+
+import java.nio.charset.StandardCharsets;
+import java.time.Instant;
+import org.apache.commons.lang.mutable.MutableBoolean;
+import org.apache.storm.Config;
+import org.apache.storm.utils.MutableObject;
+import org.apache.stormcrawler.Metadata;
+import org.apache.stormcrawler.protocol.AbstractProtocolTest;
+import org.apache.stormcrawler.protocol.ProtocolResponse;
+import org.eclipse.jetty.server.Handler;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.Timeout;
+
+/**
+ * Tests for Playwright protocol implementation. Chrome should be running on localhost, whether is
+ * has been launched manually or by Playwright.
+ */
+public class ProtocolTest extends AbstractProtocolTest {
+
+ @Rule public Timeout globalTimeout = Timeout.seconds(120);
+
+ private static final String USER_AGENT = "StormCrawlerTest";
+
+ @Override
+ protected Handler[] getHandlers() {
+ return new Handler[] {new LocalResourceHandler(), new WildcardResourceHandler()};
+ }
+
+ public HttpProtocol getProtocol() {
+
+ Config conf = new Config();
+ conf.put("http.agent.name", USER_AGENT);
+
+ // "http://localhost:9222"
+ String cdpurl = System.getProperty("playwright.cdp.url");
+ if (cdpurl != null) {
+ conf.put("playwright.cdp.url", cdpurl);
+ }
+
+ HttpProtocol protocol = new HttpProtocol();
+ protocol.configure(conf);
+ return protocol;
+ }
+
+ @Before
+ public void setup() {
+ org.junit.Assume.assumeTrue("false".equals(System.getProperty("CI_ENV", "false")));
+ }
+
+ @Test
+ public void testNotFound() throws Exception {
+ HttpProtocol protocol = getProtocol();
+ String url = "http://localhost:" + HTTP_PORT + "/doesNotExist";
+ ProtocolResponse response = protocol.getProtocolOutput(url, new Metadata());
+ Assert.assertEquals(404, response.getStatusCode());
+ Assert.assertEquals(0, response.getContent().length);
+ }
+
+ @Test
+ public void testHTML() throws Exception {
+ HttpProtocol protocol = getProtocol();
+ String url = "http://localhost:" + HTTP_PORT + "/dynamic-scraping.html";
+ ProtocolResponse response = protocol.getProtocolOutput(url, new Metadata());
+ // check that we have the metadata we expect
+ Assert.assertNotNull(response.getMetadata().getFirstValue("key"));
+ // the correct code
+ Assert.assertEquals(200, response.getStatusCode());
+
+ final String content = new String(response.getContent(), StandardCharsets.UTF_8);
+ Assert.assertNotNull(content);
+
+ // we expect that the given JS was executed, so the content should contain this HTML snippet
+ Assert.assertTrue(
+ content.contains(
+ "<p><a href=\"https://stormcrawler.apache.org/\">StormCrawler Rocks!</a></p>"));
+ }
+
+ /** Calls 2 URLs on the same protocol instance - should block * */
+ @Test
+ public void testBlocking() throws InterruptedException {
+ HttpProtocol p = getProtocol();
+ run(p, p, true);
+ }
+
+ /** Calls 2 URLs on 2 different instances - should run in parallel * */
+ @Test
+ public void testParallel() throws InterruptedException {
+ run(getProtocol(), getProtocol(), false);
+ }
+
+ private void run(HttpProtocol protocol, HttpProtocol protocol2, boolean expected)
+ throws InterruptedException {
+
+ MutableBoolean noException = new MutableBoolean(true);
+
+ MutableObject endTimeFirst = new MutableObject();
+ MutableObject startTimeSecond = new MutableObject();
+
+ String url = "http://localhost:" + HTTP_PORT + "/";
+
+ new Thread(
+ () -> {
+ try {
+ ProtocolResponse response =
+ protocol.getProtocolOutput(url, new Metadata());
+ endTimeFirst.setObject(
+ Instant.parse(
+ response.getMetadata()
+ .getFirstValue(HttpProtocol.MD_KEY_END)));
+ } catch (Exception e) {
+ noException.setValue(false);
+ }
+ })
+ .start();
+
+ new Thread(
+ () -> {
+ try {
+ ProtocolResponse response =
+ protocol2.getProtocolOutput(url, new Metadata());
+ startTimeSecond.setObject(
+ Instant.parse(
+ response.getMetadata()
+ .getFirstValue(HttpProtocol.MD_KEY_START)));
+ } catch (Exception e) {
+ noException.setValue(false);
+ }
+ })
+ .start();
+
+ while (noException.booleanValue()
+ && (endTimeFirst.getObject() == null || startTimeSecond.getObject() == null)) {
+ Thread.sleep(10);
+ }
+
+ Assert.assertEquals(true, noException.booleanValue());
+
+ Instant etf = (Instant) endTimeFirst.getObject();
+ Instant sts = (Instant) startTimeSecond.getObject();
+
+ // check that the second call started while the first one is being processed
+ // normal - dealing with different instances
+ Assert.assertEquals(expected, etf.isBefore(sts));
+
+ protocol.cleanup();
+ }
+}
diff --git a/external/playwright/src/test/resources/dynamic-scraping.html b/external/playwright/src/test/resources/dynamic-scraping.html
new file mode 100644
index 00000000..9c9d56e9
--- /dev/null
+++ b/external/playwright/src/test/resources/dynamic-scraping.html
@@ -0,0 +1,32 @@
+<!DOCTYPE html>
+<html>
+<head>
+ <title>StormCrawler Example</title>
+</head>
+<body>
+<noscript>
+ Booh booh! No JavaScript!
+</noscript>
+<script>
+ document.addEventListener("DOMContentLoaded", function() {
+ // Create a paragraph element
+ var p = document.createElement("p");
+
+ // Create an anchor (link) element
+ var a = document.createElement("a");
+
+ // Set the href attribute of the anchor element
+ a.href = "https://stormcrawler.apache.org/";
+
+ // Set the text content of the anchor element
+ a.textContent = "StormCrawler Rocks!";
+
+ // Append the anchor element to the paragraph element
+ p.appendChild(a);
+
+ // Output the paragraph element to the document body
+ document.body.appendChild(p);
+ });
+</script>
+</body>
+</html>
diff --git a/pom.xml b/pom.xml
index 61c9c8fe..b4b7b3ff 100644
--- a/pom.xml
+++ b/pom.xml
@@ -531,6 +531,7 @@ under the License.
<module>external/aws</module>
<module>external/langid</module>
<module>external/opensearch</module>
+ <module>external/playwright</module>
<module>external/solr</module>
<module>external/sql</module>
<module>external/tika</module>