You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by szaruba <st...@student.tgm.ac.at> on 2011/10/23 20:50:01 UTC

How to grab a (only one) file with camel ?

Hi everybody,

After we got the expertise, that camel doesn't support consuming a fix
amount of files and stops after all files are consumed, we want to implement
our own customized endpoint, which is able to fulfill our requirements.
 
Basically we have to grab some files at a defined moment. We got a master
file, which contains the paths to all other files. So what we have to do is
to implement an endpoint, which
a) grabs the master file and opens it
b) consumes the other files, based on the paths in the master file

Sure we could simply use java.io. to open the file and extract data with
"Classic" java. But we have to use camel in our project and we want to
support ftp and other endpoint types too. So we want to create an endpoint
which takes an URI and is able to handle more than simple file access. (just
like the normal  "from("URI)." in camel DSL) 
  
We already got a hint which belongs to the "Consumer Template", which
describes how to implement our own "pojo based" endpoint:

http://camel.465427.n5.nabble.com/Non-polling-based-file-consumer-td4888325.html

Fact is that we want to use the camel build in function, which just takes an
URI and get the files from the specified URI. I know that's not a common
question, but where do I find this component or function? Is this function
or component always coupled with a polling consumer? Anyone have an idea how
to solve our problem without finding the corresponding camel function?

Currently I am reading the "Camel in Action" book to get more into camel and
find a solution in this case, but I would be very happy if anyone could give
me a hint or can show me an example of a customized endpoint. 

If anybody need additional information please answer to this post.

Greetings and thanks in advance,
Lukas


--
View this message in context: http://camel.465427.n5.nabble.com/How-to-grab-a-only-one-file-with-camel-tp4930104p4930104.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: How to grab a (only one) file with camel ?

Posted by Jesús Castillo <je...@gmail.com>.
Hi Lukas,

I have exactly the same request. From a control file with URIs, we
need to retrieve those files from the local file
system and send them using sftp to a remote server.

The approach I intend to use is to create dynamic routes (one per file to send)
as mentioned in
http://camel.465427.n5.nabble.com/How-to-update-remove-camel-route-td478637.html
and then stop it and remove it after the file has been sent.

Regards,

Jesus.

Re: How to grab a (only one) file with camel ?

Posted by szaruba <st...@student.tgm.ac.at>.
Hi!

Thank you for your answer, but I have to tell you, that we can't use any
component specific feature, because our solution should work with file and
ftp access (Generally it should work with every component, which just needs
a URI to grab a file).

I recommend reading JB's blog to everybody who have a familiar problem.

Greetings, 
Lukas


Craig Taylor wrote:
> 
> Another alternative might be to use the File locking mechanism with
> FTP style components.
> 
> Consider the "supposed" remote lock as never released unless it's the
> proper file.   Note: When doing this, adjust your trace level
> accordingly.
> 


--
View this message in context: http://camel.465427.n5.nabble.com/How-to-grab-a-only-one-file-with-camel-tp4930104p4936295.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: How to grab a (only one) file with camel ?

Posted by Craig Taylor <ct...@ctalkobt.net>.
Another alternative might be to use the File locking mechanism with
FTP style components.

Consider the "supposed" remote lock as never released unless it's the
proper file.   Note: When doing this, adjust your trace level
accordingly.

On Sun, Oct 23, 2011 at 2:50 PM, szaruba
<st...@student.tgm.ac.at> wrote:
> Hi everybody,
>
> After we got the expertise, that camel doesn't support consuming a fix
> amount of files and stops after all files are consumed, we want to implement
> our own customized endpoint, which is able to fulfill our requirements.
>
> Basically we have to grab some files at a defined moment. We got a master
> file, which contains the paths to all other files. So what we have to do is
> to implement an endpoint, which
> a) grabs the master file and opens it
> b) consumes the other files, based on the paths in the master file
>
> Sure we could simply use java.io. to open the file and extract data with
> "Classic" java. But we have to use camel in our project and we want to
> support ftp and other endpoint types too. So we want to create an endpoint
> which takes an URI and is able to handle more than simple file access. (just
> like the normal  "from("URI)." in camel DSL)
>
> We already got a hint which belongs to the "Consumer Template", which
> describes how to implement our own "pojo based" endpoint:
>
> http://camel.465427.n5.nabble.com/Non-polling-based-file-consumer-td4888325.html
>
> Fact is that we want to use the camel build in function, which just takes an
> URI and get the files from the specified URI. I know that's not a common
> question, but where do I find this component or function? Is this function
> or component always coupled with a polling consumer? Anyone have an idea how
> to solve our problem without finding the corresponding camel function?
>
> Currently I am reading the "Camel in Action" book to get more into camel and
> find a solution in this case, but I would be very happy if anyone could give
> me a hint or can show me an example of a customized endpoint.
>
> If anybody need additional information please answer to this post.
>
> Greetings and thanks in advance,
> Lukas
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/How-to-grab-a-only-one-file-with-camel-tp4930104p4930104.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>



-- 
-------------------------------------------
Craig Taylor
ctalkobt@ctalkobt.net

Re: How to grab a (only one) file with camel ?

Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Lukas,

The Camel file component will "inject" a message per file.

It means that you can stop a route after just processing a message. You 
can control that with another route containing a bean, and define when 
you want to stop a route (after a number of processed files for instance).

In this blog:
http://blog.nanthrax.net/2011/08/use-camel-cxf-and-karaf-to-implement-batches/

I have a "login route" is stopped just after processing one message (so 
one file in the case of the file component).
You can mimic this behavior to implement your needs.

Regards
JB

On 10/23/2011 08:50 PM, szaruba wrote:
> Hi everybody,
>
> After we got the expertise, that camel doesn't support consuming a fix
> amount of files and stops after all files are consumed, we want to implement
> our own customized endpoint, which is able to fulfill our requirements.
>
> Basically we have to grab some files at a defined moment. We got a master
> file, which contains the paths to all other files. So what we have to do is
> to implement an endpoint, which
> a) grabs the master file and opens it
> b) consumes the other files, based on the paths in the master file
>
> Sure we could simply use java.io. to open the file and extract data with
> "Classic" java. But we have to use camel in our project and we want to
> support ftp and other endpoint types too. So we want to create an endpoint
> which takes an URI and is able to handle more than simple file access. (just
> like the normal  "from("URI)." in camel DSL)
>
> We already got a hint which belongs to the "Consumer Template", which
> describes how to implement our own "pojo based" endpoint:
>
> http://camel.465427.n5.nabble.com/Non-polling-based-file-consumer-td4888325.html
>
> Fact is that we want to use the camel build in function, which just takes an
> URI and get the files from the specified URI. I know that's not a common
> question, but where do I find this component or function? Is this function
> or component always coupled with a polling consumer? Anyone have an idea how
> to solve our problem without finding the corresponding camel function?
>
> Currently I am reading the "Camel in Action" book to get more into camel and
> find a solution in this case, but I would be very happy if anyone could give
> me a hint or can show me an example of a customized endpoint.
>
> If anybody need additional information please answer to this post.
>
> Greetings and thanks in advance,
> Lukas
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/How-to-grab-a-only-one-file-with-camel-tp4930104p4930104.html
> Sent from the Camel - Users mailing list archive at Nabble.com.

-- 
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: How to grab a (only one) file with camel ?

Posted by Christian Schneider <ch...@die-schneider.net>.
You could try the polling consumer.

http://camel.apache.org/polling-consumer.html

Christian


Am 23.10.2011 20:50, schrieb szaruba:
> Hi everybody,
>
> After we got the expertise, that camel doesn't support consuming a fix
> amount of files and stops after all files are consumed, we want to implement
> our own customized endpoint, which is able to fulfill our requirements.
>
> Basically we have to grab some files at a defined moment. We got a master
> file, which contains the paths to all other files. So what we have to do is
> to implement an endpoint, which
> a) grabs the master file and opens it
> b) consumes the other files, based on the paths in the master file
>
> Sure we could simply use java.io. to open the file and extract data with
> "Classic" java. But we have to use camel in our project and we want to
> support ftp and other endpoint types too. So we want to create an endpoint
> which takes an URI and is able to handle more than simple file access. (just
> like the normal  "from("URI)." in camel DSL)
>
> We already got a hint which belongs to the "Consumer Template", which
> describes how to implement our own "pojo based" endpoint:
>
> http://camel.465427.n5.nabble.com/Non-polling-based-file-consumer-td4888325.html
>
> Fact is that we want to use the camel build in function, which just takes an
> URI and get the files from the specified URI. I know that's not a common
> question, but where do I find this component or function? Is this function
> or component always coupled with a polling consumer? Anyone have an idea how
> to solve our problem without finding the corresponding camel function?
>
> Currently I am reading the "Camel in Action" book to get more into camel and
> find a solution in this case, but I would be very happy if anyone could give
> me a hint or can show me an example of a customized endpoint.
>
> If anybody need additional information please answer to this post.
>
> Greetings and thanks in advance,
> Lukas
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/How-to-grab-a-only-one-file-with-camel-tp4930104p4930104.html
> Sent from the Camel - Users mailing list archive at Nabble.com.


-- 

Christian Schneider
http://www.liquid-reality.de

Open Source Architect
Talend Application Integration Division http://www.talend.com


Re: How to grab a (only one) file with camel ?

Posted by edvicif <ed...@gmail.com>.
I have a solution. Not sure this is the final form.

Dynamic Router Factory


I've created a DynamicRouteBuilder. Which dynamically create a route, with a
direct entry point and a configurable exit point.

class DynamicRouteBuilder extends RouteBuilder{
  
  private final String enrichUri;
  private final String outUri;
  private final String routeId;
  private static final AggregationStrategy MY_AGGREGATION_STRATEGY = new
MyAggregationStrategy();

  public DynamicRouteBuilder(String enrichUri, String outUri, String
routeId) {
    this.enrichUri = enrichUri;
    this.outUri = outUri;
    this.routeId = routeId;
  }

  //Dynamic route creation
  @Override
  public void configure() throws Exception {
    from("direct:"+routeId).id(routeId) //Dynamic entry point
      .pollEnrich(enrichUri, 2000, MY_AGGREGATION_STRATEGY)
      .to(outUri) //Exit point, where we can wait staically to continue
processing
      .process(new *ShutDownProcessor*());
  }
  
  //Route shutdown it self after completion
  private final class *ShutDownProcessor* implements Processor {
    @Override
    public void process(Exchange exchange) throws Exception {
      CamelContext camelContext = exchange.getContext();
      camelContext.getInflightRepository().remove(exchange);
      /camelContext.stopRoute(routeId);
      camelContext.removeRoute(routeId);/
    }
  }
  ... Aggregation Strategy implemetation left off to spare some space
}

The router expects a name, which will be served as an entry point to this
route. We enrich it, than forwardingit to an existing static route.

After sending out the message basically we call the Shutdown processor,
which destroys the route. So basically the route exist for one message.
There could be an issue if because async we send multiple message. But that
is why routeId is configruable. Maybe with combining 
http://camel.apache.org/dynamic-router.html dynamic router EIP  it can be
solved.

Invoking Dynamic Route


 from("direct:fileCopier")
            .process(new Processor() {
              @Override
              public void process(Exchange exchange) throws Exception {
                String fileName = exchange.getIn().getBody(String.class);
                DynamicRouteBuilder dynamicRouteBuilder = new
DynamicRouteBuilder("file:target/test-classes/fileTest?fileName=" +
fileName, 
                    "direct:*eventFileProducer*",
                    "*eventFilePooler*"); //dynamic route name
                exchange.getContext().addRoutes(dynamicRouteBuilder);
              }
            })
            .to("direct:*eventFilePooler*"); //sending the message to the
dynamic route

The fileCopier route just creates the dynamic route with a specific direct
endpoint. It specify the file, what he want to consume using enpoint uri and
where I'd like to route the results (direct:eventFileProducer)

Exitpoint of Dynamic Route


So from now on I have a statical route to continue the processing
from("direct:*eventFileProducer*").to("log:doEverything as usual");

I don't know this is what we are looking for.

--
View this message in context: http://camel.465427.n5.nabble.com/How-to-grab-a-only-one-file-with-camel-tp4930104p5508497.html
Sent from the Camel - Users mailing list archive at Nabble.com.