You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by "oscerd (via GitHub)" <gi...@apache.org> on 2025/10/23 11:37:08 UTC
[PR] CAMEL-22592 - Camel-Docling: Support Batch operations [camel]
oscerd opened a new pull request, #19676:
URL: https://github.com/apache/camel/pull/19676
# Description
<!--
- Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
-->
# Target
- [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch)
# Tracking
- [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it).
<!--
# *Note*: trivial changes like, typos, minor documentation fixes and other small items do not require a JIRA issue. In this case your pull request should address just this issue, without pulling in other changes.
-->
# Apache Camel coding standards and style
- [x] I checked that each commit in the pull request has a meaningful subject line and body.
<!--
If you're unsure, you can format the pull request title like `[CAMEL-XXX] Fixes bug in camel-file component`, where you replace `CAMEL-XXX` with the appropriate JIRA issue.
-->
- [x] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes.
<!--
You can run the aforementioned command in your module so that the build auto-formats your code. This will also be verified as part of the checks and your PR may be rejected if if there are uncommited changes after running `mvn clean install -DskipTests`.
You can learn more about the contribution guidelines at https://github.com/apache/camel/blob/main/CONTRIBUTING.md
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-22592 - Camel-Docling: Support Batch operations [camel]
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on code in PR #19676:
URL: https://github.com/apache/camel/pull/19676#discussion_r2456241890
##########
components/camel-ai/camel-docling/src/main/java/org/apache/camel/component/docling/DoclingServeClient.java:
##########
@@ -653,6 +662,124 @@ private String fetchTaskResult(String taskId) throws IOException {
}
}
+ /**
+ * Convert multiple documents in batch using parallel processing.
+ *
+ * @param inputSources List of file paths or URLs to documents
+ * @param outputFormat Output format (md, json, html, text)
+ * @param batchSize Maximum number of documents in this batch
+ * @param parallelism Number of parallel threads to use
+ * @param failOnFirstError Whether to fail entire batch on first error
+ * @param useAsync Whether to use async mode for individual conversions
+ * @return BatchProcessingResults containing all conversion results
+ */
+ public BatchProcessingResults convertDocumentsBatch(
+ List<String> inputSources, String outputFormat, int batchSize, int parallelism,
+ boolean failOnFirstError, boolean useAsync) {
+
+ LOG.info("Starting batch conversion of {} documents with parallelism={}, failOnFirstError={}",
+ inputSources.size(), parallelism, failOnFirstError);
+
+ BatchProcessingResults results = new BatchProcessingResults();
+ results.setStartTimeMs(System.currentTimeMillis());
+
+ ExecutorService executor = Executors.newFixedThreadPool(parallelism);
+ List<Future<BatchConversionResult>> futures = new ArrayList<>();
+ AtomicInteger index = new AtomicInteger(0);
+
+ try {
+ // Submit all conversion tasks
+ for (String inputSource : inputSources) {
+ final int currentIndex = index.getAndIncrement();
+ final String documentId = "doc-" + currentIndex;
+
+ Callable<BatchConversionResult> task = () -> {
+ BatchConversionResult result = new BatchConversionResult(documentId, inputSource);
+ result.setBatchIndex(currentIndex);
+ long startTime = System.currentTimeMillis();
+
+ try {
+ LOG.debug("Processing document {} (index {}): {}", documentId, currentIndex, inputSource);
+
+ String converted;
+ if (useAsync) {
+ converted = convertDocumentAsyncAndWait(inputSource, outputFormat);
+ } else {
+ converted = convertDocument(inputSource, outputFormat);
+ }
+
+ result.setResult(converted);
+ result.setSuccess(true);
+ result.setProcessingTimeMs(System.currentTimeMillis() - startTime);
+
+ LOG.debug("Successfully processed document {} in {}ms", documentId, result.getProcessingTimeMs());
+
+ } catch (Exception e) {
+ result.setSuccess(false);
+ result.setErrorMessage(e.getMessage());
+ result.setProcessingTimeMs(System.currentTimeMillis() - startTime);
+
+ LOG.error("Failed to process document {} (index {}): {}", documentId, currentIndex,
+ e.getMessage(), e);
+ }
+
+ return result;
+ };
+
+ futures.add(executor.submit(task));
+ }
+
+ // Collect results
+ for (Future<BatchConversionResult> future : futures) {
+ try {
+ BatchConversionResult result = future.get();
Review Comment:
I refactored a bit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-22592 - Camel-Docling: Support Batch operations [camel]
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on PR #19676:
URL: https://github.com/apache/camel/pull/19676#issuecomment-3441515616
This is failing because on gh actions the docling-serve image is too big.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-22592 - Camel-Docling: Support Batch operations [camel]
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on code in PR #19676:
URL: https://github.com/apache/camel/pull/19676#discussion_r2454991098
##########
components/camel-ai/camel-docling/src/main/java/org/apache/camel/component/docling/DoclingServeClient.java:
##########
@@ -653,6 +662,124 @@ private String fetchTaskResult(String taskId) throws IOException {
}
}
+ /**
+ * Convert multiple documents in batch using parallel processing.
+ *
+ * @param inputSources List of file paths or URLs to documents
+ * @param outputFormat Output format (md, json, html, text)
+ * @param batchSize Maximum number of documents in this batch
+ * @param parallelism Number of parallel threads to use
+ * @param failOnFirstError Whether to fail entire batch on first error
+ * @param useAsync Whether to use async mode for individual conversions
+ * @return BatchProcessingResults containing all conversion results
+ */
+ public BatchProcessingResults convertDocumentsBatch(
+ List<String> inputSources, String outputFormat, int batchSize, int parallelism,
+ boolean failOnFirstError, boolean useAsync) {
+
+ LOG.info("Starting batch conversion of {} documents with parallelism={}, failOnFirstError={}",
+ inputSources.size(), parallelism, failOnFirstError);
+
+ BatchProcessingResults results = new BatchProcessingResults();
+ results.setStartTimeMs(System.currentTimeMillis());
+
+ ExecutorService executor = Executors.newFixedThreadPool(parallelism);
+ List<Future<BatchConversionResult>> futures = new ArrayList<>();
+ AtomicInteger index = new AtomicInteger(0);
+
+ try {
+ // Submit all conversion tasks
+ for (String inputSource : inputSources) {
+ final int currentIndex = index.getAndIncrement();
+ final String documentId = "doc-" + currentIndex;
+
+ Callable<BatchConversionResult> task = () -> {
+ BatchConversionResult result = new BatchConversionResult(documentId, inputSource);
+ result.setBatchIndex(currentIndex);
+ long startTime = System.currentTimeMillis();
+
+ try {
+ LOG.debug("Processing document {} (index {}): {}", documentId, currentIndex, inputSource);
+
+ String converted;
+ if (useAsync) {
+ converted = convertDocumentAsyncAndWait(inputSource, outputFormat);
+ } else {
+ converted = convertDocument(inputSource, outputFormat);
+ }
+
+ result.setResult(converted);
+ result.setSuccess(true);
+ result.setProcessingTimeMs(System.currentTimeMillis() - startTime);
+
+ LOG.debug("Successfully processed document {} in {}ms", documentId, result.getProcessingTimeMs());
+
+ } catch (Exception e) {
+ result.setSuccess(false);
+ result.setErrorMessage(e.getMessage());
+ result.setProcessingTimeMs(System.currentTimeMillis() - startTime);
+
+ LOG.error("Failed to process document {} (index {}): {}", documentId, currentIndex,
+ e.getMessage(), e);
+ }
+
+ return result;
+ };
+
+ futures.add(executor.submit(task));
+ }
+
+ // Collect results
+ for (Future<BatchConversionResult> future : futures) {
+ try {
+ BatchConversionResult result = future.get();
Review Comment:
Yes, this is something I missed in the first place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-22592 - Camel-Docling: Support Batch operations [camel]
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd merged PR #19676:
URL: https://github.com/apache/camel/pull/19676
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-22592 - Camel-Docling: Support Batch operations [camel]
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on code in PR #19676:
URL: https://github.com/apache/camel/pull/19676#discussion_r2454993341
##########
components/camel-ai/camel-docling/src/main/java/org/apache/camel/component/docling/DoclingServeClient.java:
##########
@@ -653,6 +662,124 @@ private String fetchTaskResult(String taskId) throws IOException {
}
}
+ /**
+ * Convert multiple documents in batch using parallel processing.
+ *
+ * @param inputSources List of file paths or URLs to documents
+ * @param outputFormat Output format (md, json, html, text)
+ * @param batchSize Maximum number of documents in this batch
+ * @param parallelism Number of parallel threads to use
+ * @param failOnFirstError Whether to fail entire batch on first error
+ * @param useAsync Whether to use async mode for individual conversions
+ * @return BatchProcessingResults containing all conversion results
+ */
+ public BatchProcessingResults convertDocumentsBatch(
+ List<String> inputSources, String outputFormat, int batchSize, int parallelism,
+ boolean failOnFirstError, boolean useAsync) {
+
+ LOG.info("Starting batch conversion of {} documents with parallelism={}, failOnFirstError={}",
+ inputSources.size(), parallelism, failOnFirstError);
+
+ BatchProcessingResults results = new BatchProcessingResults();
+ results.setStartTimeMs(System.currentTimeMillis());
+
+ ExecutorService executor = Executors.newFixedThreadPool(parallelism);
+ List<Future<BatchConversionResult>> futures = new ArrayList<>();
+ AtomicInteger index = new AtomicInteger(0);
+
+ try {
+ // Submit all conversion tasks
+ for (String inputSource : inputSources) {
+ final int currentIndex = index.getAndIncrement();
+ final String documentId = "doc-" + currentIndex;
+
+ Callable<BatchConversionResult> task = () -> {
+ BatchConversionResult result = new BatchConversionResult(documentId, inputSource);
+ result.setBatchIndex(currentIndex);
+ long startTime = System.currentTimeMillis();
+
+ try {
+ LOG.debug("Processing document {} (index {}): {}", documentId, currentIndex, inputSource);
+
+ String converted;
+ if (useAsync) {
+ converted = convertDocumentAsyncAndWait(inputSource, outputFormat);
+ } else {
+ converted = convertDocument(inputSource, outputFormat);
+ }
+
+ result.setResult(converted);
+ result.setSuccess(true);
+ result.setProcessingTimeMs(System.currentTimeMillis() - startTime);
+
+ LOG.debug("Successfully processed document {} in {}ms", documentId, result.getProcessingTimeMs());
+
+ } catch (Exception e) {
+ result.setSuccess(false);
+ result.setErrorMessage(e.getMessage());
+ result.setProcessingTimeMs(System.currentTimeMillis() - startTime);
+
+ LOG.error("Failed to process document {} (index {}): {}", documentId, currentIndex,
+ e.getMessage(), e);
+ }
+
+ return result;
+ };
+
+ futures.add(executor.submit(task));
+ }
+
+ // Collect results
+ for (Future<BatchConversionResult> future : futures) {
+ try {
+ BatchConversionResult result = future.get();
Review Comment:
You're right, I didn't think about the case of failOnFirstError. I'll update this one too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-22592 - Camel-Docling: Support Batch operations [camel]
Posted by "apupier (via GitHub)" <gi...@apache.org>.
apupier commented on code in PR #19676:
URL: https://github.com/apache/camel/pull/19676#discussion_r2454908438
##########
components/camel-ai/camel-docling/src/main/java/org/apache/camel/component/docling/DoclingProducer.java:
##########
@@ -261,6 +278,164 @@ private void processCheckConversionStatus(Exchange exchange) throws Exception {
}
}
+ private void processBatchConversion(Exchange exchange, String outputFormat) throws Exception {
+ LOG.debug("DoclingProducer processing batch conversion with format: {}", outputFormat);
+
+ if (!configuration.isUseDoclingServe()) {
+ throw new IllegalStateException(
+ "Batch operations require docling-serve mode (useDoclingServe=true)");
+ }
+
+ // Extract document list from body
+ List<String> documentPaths = extractDocumentList(exchange);
+
+ if (documentPaths.isEmpty()) {
+ throw new IllegalArgumentException("No documents provided for batch processing");
+ }
+
+ LOG.debug("Processing batch of {} documents", documentPaths.size());
+
+ // Get batch configuration from headers or use defaults
+ int batchSize = exchange.getIn().getHeader(DoclingHeaders.BATCH_SIZE, configuration.getBatchSize(), Integer.class);
+ int parallelism
+ = exchange.getIn().getHeader(DoclingHeaders.BATCH_PARALLELISM, configuration.getBatchParallelism(),
+ Integer.class);
+ boolean failOnFirstError = exchange.getIn().getHeader(DoclingHeaders.BATCH_FAIL_ON_FIRST_ERROR,
+ configuration.isBatchFailOnFirstError(), Boolean.class);
+
+ // Check if we should use async mode for individual conversions
+ boolean useAsync = configuration.isUseAsyncMode();
+ Boolean asyncModeHeader = exchange.getIn().getHeader(DoclingHeaders.USE_ASYNC_MODE, Boolean.class);
+ if (asyncModeHeader != null) {
+ useAsync = asyncModeHeader;
+ }
+
+ // Process batch using DoclingServeClient
+ BatchProcessingResults results = doclingServeClient.convertDocumentsBatch(
+ documentPaths, outputFormat, batchSize, parallelism, failOnFirstError, useAsync);
+
+ // Check if we should split results into individual exchanges
+ boolean splitResults = configuration.isSplitBatchResults();
+ Boolean splitResultsHeader = exchange.getIn().getHeader("CamelDoclingBatchSplitResults", Boolean.class);
Review Comment:
Why this one is not part of DoclingHeaders?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-22592 - Camel-Docling: Support Batch operations [camel]
Posted by "apupier (via GitHub)" <gi...@apache.org>.
apupier commented on code in PR #19676:
URL: https://github.com/apache/camel/pull/19676#discussion_r2454941116
##########
components/camel-ai/camel-docling/src/main/java/org/apache/camel/component/docling/DoclingServeClient.java:
##########
@@ -653,6 +662,124 @@ private String fetchTaskResult(String taskId) throws IOException {
}
}
+ /**
+ * Convert multiple documents in batch using parallel processing.
+ *
+ * @param inputSources List of file paths or URLs to documents
+ * @param outputFormat Output format (md, json, html, text)
+ * @param batchSize Maximum number of documents in this batch
+ * @param parallelism Number of parallel threads to use
+ * @param failOnFirstError Whether to fail entire batch on first error
+ * @param useAsync Whether to use async mode for individual conversions
+ * @return BatchProcessingResults containing all conversion results
+ */
+ public BatchProcessingResults convertDocumentsBatch(
+ List<String> inputSources, String outputFormat, int batchSize, int parallelism,
+ boolean failOnFirstError, boolean useAsync) {
+
+ LOG.info("Starting batch conversion of {} documents with parallelism={}, failOnFirstError={}",
+ inputSources.size(), parallelism, failOnFirstError);
+
+ BatchProcessingResults results = new BatchProcessingResults();
+ results.setStartTimeMs(System.currentTimeMillis());
+
+ ExecutorService executor = Executors.newFixedThreadPool(parallelism);
+ List<Future<BatchConversionResult>> futures = new ArrayList<>();
+ AtomicInteger index = new AtomicInteger(0);
+
+ try {
+ // Submit all conversion tasks
+ for (String inputSource : inputSources) {
+ final int currentIndex = index.getAndIncrement();
+ final String documentId = "doc-" + currentIndex;
+
+ Callable<BatchConversionResult> task = () -> {
+ BatchConversionResult result = new BatchConversionResult(documentId, inputSource);
+ result.setBatchIndex(currentIndex);
+ long startTime = System.currentTimeMillis();
+
+ try {
+ LOG.debug("Processing document {} (index {}): {}", documentId, currentIndex, inputSource);
+
+ String converted;
+ if (useAsync) {
+ converted = convertDocumentAsyncAndWait(inputSource, outputFormat);
+ } else {
+ converted = convertDocument(inputSource, outputFormat);
+ }
+
+ result.setResult(converted);
+ result.setSuccess(true);
+ result.setProcessingTimeMs(System.currentTimeMillis() - startTime);
+
+ LOG.debug("Successfully processed document {} in {}ms", documentId, result.getProcessingTimeMs());
+
+ } catch (Exception e) {
+ result.setSuccess(false);
+ result.setErrorMessage(e.getMessage());
+ result.setProcessingTimeMs(System.currentTimeMillis() - startTime);
+
+ LOG.error("Failed to process document {} (index {}): {}", documentId, currentIndex,
+ e.getMessage(), e);
+ }
+
+ return result;
+ };
+
+ futures.add(executor.submit(task));
+ }
+
+ // Collect results
+ for (Future<BatchConversionResult> future : futures) {
+ try {
+ BatchConversionResult result = future.get();
Review Comment:
When i read that it make me think about CompletableFuture.allOf() (at least in case failOnFirstError).Have you already looked to this possibility? And if yes, what is the reason to not use it?
(absolutely not blocking even if the answer is no, this can still be looked at later)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-22592 - Camel-Docling: Support Batch operations [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on code in PR #19676:
URL: https://github.com/apache/camel/pull/19676#discussion_r2454898304
##########
components/camel-ai/camel-docling/src/main/java/org/apache/camel/component/docling/DoclingServeClient.java:
##########
@@ -653,6 +662,124 @@ private String fetchTaskResult(String taskId) throws IOException {
}
}
+ /**
+ * Convert multiple documents in batch using parallel processing.
+ *
+ * @param inputSources List of file paths or URLs to documents
+ * @param outputFormat Output format (md, json, html, text)
+ * @param batchSize Maximum number of documents in this batch
+ * @param parallelism Number of parallel threads to use
+ * @param failOnFirstError Whether to fail entire batch on first error
+ * @param useAsync Whether to use async mode for individual conversions
+ * @return BatchProcessingResults containing all conversion results
+ */
+ public BatchProcessingResults convertDocumentsBatch(
+ List<String> inputSources, String outputFormat, int batchSize, int parallelism,
+ boolean failOnFirstError, boolean useAsync) {
+
+ LOG.info("Starting batch conversion of {} documents with parallelism={}, failOnFirstError={}",
+ inputSources.size(), parallelism, failOnFirstError);
+
+ BatchProcessingResults results = new BatchProcessingResults();
+ results.setStartTimeMs(System.currentTimeMillis());
+
+ ExecutorService executor = Executors.newFixedThreadPool(parallelism);
+ List<Future<BatchConversionResult>> futures = new ArrayList<>();
+ AtomicInteger index = new AtomicInteger(0);
+
+ try {
+ // Submit all conversion tasks
+ for (String inputSource : inputSources) {
+ final int currentIndex = index.getAndIncrement();
+ final String documentId = "doc-" + currentIndex;
+
+ Callable<BatchConversionResult> task = () -> {
+ BatchConversionResult result = new BatchConversionResult(documentId, inputSource);
+ result.setBatchIndex(currentIndex);
+ long startTime = System.currentTimeMillis();
+
+ try {
+ LOG.debug("Processing document {} (index {}): {}", documentId, currentIndex, inputSource);
+
+ String converted;
+ if (useAsync) {
+ converted = convertDocumentAsyncAndWait(inputSource, outputFormat);
+ } else {
+ converted = convertDocument(inputSource, outputFormat);
+ }
+
+ result.setResult(converted);
+ result.setSuccess(true);
+ result.setProcessingTimeMs(System.currentTimeMillis() - startTime);
+
+ LOG.debug("Successfully processed document {} in {}ms", documentId, result.getProcessingTimeMs());
+
+ } catch (Exception e) {
+ result.setSuccess(false);
+ result.setErrorMessage(e.getMessage());
+ result.setProcessingTimeMs(System.currentTimeMillis() - startTime);
+
+ LOG.error("Failed to process document {} (index {}): {}", documentId, currentIndex,
+ e.getMessage(), e);
+ }
+
+ return result;
+ };
+
+ futures.add(executor.submit(task));
+ }
+
+ // Collect results
+ for (Future<BatchConversionResult> future : futures) {
+ try {
+ BatchConversionResult result = future.get();
Review Comment:
This get will block, I think you should use the batchTimeout here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-22592 - Camel-Docling: Support Batch operations [camel]
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on PR #19676:
URL: https://github.com/apache/camel/pull/19676#issuecomment-3436715391
There is still some work to be done, let me convert to draft
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-22592 - Camel-Docling: Support Batch operations [camel]
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on code in PR #19676:
URL: https://github.com/apache/camel/pull/19676#discussion_r2456240729
##########
components/camel-ai/camel-docling/src/main/java/org/apache/camel/component/docling/DoclingServeClient.java:
##########
@@ -653,6 +662,124 @@ private String fetchTaskResult(String taskId) throws IOException {
}
}
+ /**
+ * Convert multiple documents in batch using parallel processing.
+ *
+ * @param inputSources List of file paths or URLs to documents
+ * @param outputFormat Output format (md, json, html, text)
+ * @param batchSize Maximum number of documents in this batch
+ * @param parallelism Number of parallel threads to use
+ * @param failOnFirstError Whether to fail entire batch on first error
+ * @param useAsync Whether to use async mode for individual conversions
+ * @return BatchProcessingResults containing all conversion results
+ */
+ public BatchProcessingResults convertDocumentsBatch(
+ List<String> inputSources, String outputFormat, int batchSize, int parallelism,
+ boolean failOnFirstError, boolean useAsync) {
+
+ LOG.info("Starting batch conversion of {} documents with parallelism={}, failOnFirstError={}",
+ inputSources.size(), parallelism, failOnFirstError);
+
+ BatchProcessingResults results = new BatchProcessingResults();
+ results.setStartTimeMs(System.currentTimeMillis());
+
+ ExecutorService executor = Executors.newFixedThreadPool(parallelism);
+ List<Future<BatchConversionResult>> futures = new ArrayList<>();
+ AtomicInteger index = new AtomicInteger(0);
+
+ try {
+ // Submit all conversion tasks
+ for (String inputSource : inputSources) {
+ final int currentIndex = index.getAndIncrement();
+ final String documentId = "doc-" + currentIndex;
+
+ Callable<BatchConversionResult> task = () -> {
+ BatchConversionResult result = new BatchConversionResult(documentId, inputSource);
+ result.setBatchIndex(currentIndex);
+ long startTime = System.currentTimeMillis();
+
+ try {
+ LOG.debug("Processing document {} (index {}): {}", documentId, currentIndex, inputSource);
+
+ String converted;
+ if (useAsync) {
+ converted = convertDocumentAsyncAndWait(inputSource, outputFormat);
+ } else {
+ converted = convertDocument(inputSource, outputFormat);
+ }
+
+ result.setResult(converted);
+ result.setSuccess(true);
+ result.setProcessingTimeMs(System.currentTimeMillis() - startTime);
+
+ LOG.debug("Successfully processed document {} in {}ms", documentId, result.getProcessingTimeMs());
+
+ } catch (Exception e) {
+ result.setSuccess(false);
+ result.setErrorMessage(e.getMessage());
+ result.setProcessingTimeMs(System.currentTimeMillis() - startTime);
+
+ LOG.error("Failed to process document {} (index {}): {}", documentId, currentIndex,
+ e.getMessage(), e);
+ }
+
+ return result;
+ };
+
+ futures.add(executor.submit(task));
+ }
+
+ // Collect results
+ for (Future<BatchConversionResult> future : futures) {
+ try {
+ BatchConversionResult result = future.get();
Review Comment:
It should be better now.
##########
components/camel-ai/camel-docling/src/main/java/org/apache/camel/component/docling/DoclingProducer.java:
##########
@@ -261,6 +278,164 @@ private void processCheckConversionStatus(Exchange exchange) throws Exception {
}
}
+ private void processBatchConversion(Exchange exchange, String outputFormat) throws Exception {
+ LOG.debug("DoclingProducer processing batch conversion with format: {}", outputFormat);
+
+ if (!configuration.isUseDoclingServe()) {
+ throw new IllegalStateException(
+ "Batch operations require docling-serve mode (useDoclingServe=true)");
+ }
+
+ // Extract document list from body
+ List<String> documentPaths = extractDocumentList(exchange);
+
+ if (documentPaths.isEmpty()) {
+ throw new IllegalArgumentException("No documents provided for batch processing");
+ }
+
+ LOG.debug("Processing batch of {} documents", documentPaths.size());
+
+ // Get batch configuration from headers or use defaults
+ int batchSize = exchange.getIn().getHeader(DoclingHeaders.BATCH_SIZE, configuration.getBatchSize(), Integer.class);
+ int parallelism
+ = exchange.getIn().getHeader(DoclingHeaders.BATCH_PARALLELISM, configuration.getBatchParallelism(),
+ Integer.class);
+ boolean failOnFirstError = exchange.getIn().getHeader(DoclingHeaders.BATCH_FAIL_ON_FIRST_ERROR,
+ configuration.isBatchFailOnFirstError(), Boolean.class);
+
+ // Check if we should use async mode for individual conversions
+ boolean useAsync = configuration.isUseAsyncMode();
+ Boolean asyncModeHeader = exchange.getIn().getHeader(DoclingHeaders.USE_ASYNC_MODE, Boolean.class);
+ if (asyncModeHeader != null) {
+ useAsync = asyncModeHeader;
+ }
+
+ // Process batch using DoclingServeClient
+ BatchProcessingResults results = doclingServeClient.convertDocumentsBatch(
+ documentPaths, outputFormat, batchSize, parallelism, failOnFirstError, useAsync);
+
+ // Check if we should split results into individual exchanges
+ boolean splitResults = configuration.isSplitBatchResults();
+ Boolean splitResultsHeader = exchange.getIn().getHeader("CamelDoclingBatchSplitResults", Boolean.class);
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-22592 - Camel-Docling: Support Batch operations [camel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19676:
URL: https://github.com/apache/camel/pull/19676#issuecomment-3436475396
:star2: Thank you for your contribution to the Apache Camel project! :star2:
:robot: CI automation will test this PR automatically.
:camel: Apache Camel Committers, please review the following items:
* First-time contributors **require MANUAL approval** for the GitHub Actions to run
* You can use the command `/component-test (camel-)component-name1 (camel-)component-name2..` to request a test from the test bot.
* You can label PRs using `build-all`, `build-dependents`, `skip-tests` and `test-dependents` to fine-tune the checks executed by this PR.
* Build and test logs are available in the Summary page. **Only** [Apache Camel committers](https://camel.apache.org/community/team/#committers) have access to the summary.
* :warning: Be careful when sharing logs. Review their contents before sharing them publicly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-22592 - Camel-Docling: Support Batch operations [camel]
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on code in PR #19676:
URL: https://github.com/apache/camel/pull/19676#discussion_r2454989715
##########
components/camel-ai/camel-docling/src/main/java/org/apache/camel/component/docling/DoclingProducer.java:
##########
@@ -261,6 +278,164 @@ private void processCheckConversionStatus(Exchange exchange) throws Exception {
}
}
+ private void processBatchConversion(Exchange exchange, String outputFormat) throws Exception {
+ LOG.debug("DoclingProducer processing batch conversion with format: {}", outputFormat);
+
+ if (!configuration.isUseDoclingServe()) {
+ throw new IllegalStateException(
+ "Batch operations require docling-serve mode (useDoclingServe=true)");
+ }
+
+ // Extract document list from body
+ List<String> documentPaths = extractDocumentList(exchange);
+
+ if (documentPaths.isEmpty()) {
+ throw new IllegalArgumentException("No documents provided for batch processing");
+ }
+
+ LOG.debug("Processing batch of {} documents", documentPaths.size());
+
+ // Get batch configuration from headers or use defaults
+ int batchSize = exchange.getIn().getHeader(DoclingHeaders.BATCH_SIZE, configuration.getBatchSize(), Integer.class);
+ int parallelism
+ = exchange.getIn().getHeader(DoclingHeaders.BATCH_PARALLELISM, configuration.getBatchParallelism(),
+ Integer.class);
+ boolean failOnFirstError = exchange.getIn().getHeader(DoclingHeaders.BATCH_FAIL_ON_FIRST_ERROR,
+ configuration.isBatchFailOnFirstError(), Boolean.class);
+
+ // Check if we should use async mode for individual conversions
+ boolean useAsync = configuration.isUseAsyncMode();
+ Boolean asyncModeHeader = exchange.getIn().getHeader(DoclingHeaders.USE_ASYNC_MODE, Boolean.class);
+ if (asyncModeHeader != null) {
+ useAsync = asyncModeHeader;
+ }
+
+ // Process batch using DoclingServeClient
+ BatchProcessingResults results = doclingServeClient.convertDocumentsBatch(
+ documentPaths, outputFormat, batchSize, parallelism, failOnFirstError, useAsync);
+
+ // Check if we should split results into individual exchanges
+ boolean splitResults = configuration.isSplitBatchResults();
+ Boolean splitResultsHeader = exchange.getIn().getHeader("CamelDoclingBatchSplitResults", Boolean.class);
Review Comment:
This is the Intellij IDE which didn't substitute that with a constant as it was suggesting... I'll update that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org