how to store values into some persistent object when using File Endpoints

[Thomas Humphries <th...@heliase.com>]

I have checked the docs, forums and FAQs but I can't find a way to do what
I need.

We have SpringBoot+Camel RESTful service:
Camel Core 2.22.1, Spring Boot starter 2.22.1 under Spring Boot 2.0.3.

We have the following (slightly obfuscated for IP reasons) rest routes to
receive, process and return a file:

*RouteBuilder*.java
...
@Override
public void configure() throws Exception {
from("file:func/input")
.unmarshal().json(JsonLibrary.Jackson, Options.class)
.process(optionsToHeaders)
.process(functionEngineProcessor)
.to("file:func/output?tempPrefix=.inprogress/");
...

*FileReceiverRouteBuilder*.java
...
@Override
public void configure() throws Exception {
rest("/func")
.post("/process")
.consumes(MediaType.TEXT_PLAIN_VALUE)
.route().routeId("func-upload-route")
.setHeader(Exchange.FILE_NAME, () -> UUID.randomUUID().toString())  // use
a UUID for inprocess filename
.to("file:uploads")
   .setBody((ex) -> buildOptions(ex))   // appends  json as function.options
   .marshal().json(JsonLibrary.Jackson, true)
   .log(LoggingLevel.INFO, "function.options: ${in.body}")
   .to("file:func/input")
   .setBody((ex) -> new ProcessResponse((String)
ex.getIn().getHeader(Exchange.FILE_NAME)))
   .marshal().json(JsonLibrary.Jackson, true);

rest("/func")
.get("status/{token}")
.route().routeId("get-generated-func-file")
.choice()
.when((ex) -> Paths.get("function/output", (String)
ex.getIn().getHeader("token")).toFile().exists())
.to("direct:func-return-file-content")
.otherwise()
.setHeader(Exchange.HTTP_RESPONSE_CODE, constant("202"));    /// send 202
until file complete

from("direct:func-return-file-content")
.log("returning file content")
.setBody((ex) -> Paths.get("func/output", (String)
ex.getIn().getHeader("token")).toFile())
.setHeader(Exchange.CONTENT_TYPE, constant(MediaType.TEXT_PLAIN_VALUE));
...


The *OptionsToHeadersProcessor* puts a filename and the Config options into
json.

public class OptionsToHeadersProcessor implements Processor{

public static final String PROCESS_PROFILE = "PROCESS_PROFILE";
public static final String FILE_NAME = "FILE_NAME";

@Override
public void process(Exchange exchange) throws Exception {
Options options = exchange.getIn().getBody(Options.class);
exchange.getIn().setHeader(FILE_NAME, options.getInputFile());
exchange.getIn().setHeader(PROCESS_PROFILE, options.getConfiguration());
 //as header

}

The processing is working fine, files are returning etc.  We have log4j
performing log appending (rollingfile) for the service, also OK.

However, we now want to enhance by generating a statistical set of data
about each file processed.
Statistics required include: -

   -  process.start time, end time and duration,
   -  one or more method timings - also start, end dutration (from within
   the main *functionEngineProcessor*)
   -  data about the composition of the input file (e.g. bytes/lines) in
   total, and for specific sections within the file.


We also need to compile all those stats into a *single (json) text* - a
single line of json - before we emit to a file.

This last requirement is causing the most issue; I think you can see that
if I need to put, say start, end time and duration together to log them
(even using log4j), I need to persist the start time value across the
processors, so that my statistics processor can calculate the final end
time and duration.

I have tried using an intercept, just before the first endpoint definition
in  *FileReceiverRouteBuilder*.java:

interceptSendToEndpoint("direct:func*")
.log(LoggingLevel.INFO, "intercepted send to output for metrics")
.process(metricsProcessor);

However *metricsProcessor* cannot retrieve anything from headers (not even
PROCESS_PROFILE or FILE_NAME) since apparently file endpints do not persist
properties or headers.

The "token" is retrievable, but that is tied to the client REST signature
and I cant extend that without a lot of disruption.  I can derive a
filename from the token, so processor is able to read the input and output
files and gather some of the required metrics.  I would rather use/derive
from FILE_NAME, if it were available.

I therefore am unsure how to store values into some persistent object for
retreival / use in the metrics processor, since passing properties and
headers between routes isn't available with File EndPoints.

My current reading seems to imply JMS is a way to achieve this? I'm new to
camel etc and I wanted to see if I'm missing something obvious.

We would rather not propose any downstream concatenation if possible.

Any help much appreciated!






-- 

*Thomas Humphries*

<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Virus-free.
www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>