You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by lassesvestergaard <la...@gmail.com> on 2013/09/02 10:16:17 UTC
Change timer runtime
Hi all.
I have a problem regarding changing the timer period on the fly. My concrete
problem is that I have around 1.200.000 historic records I need to extract
at one time, and the database table is updated every 24 hours. Furthermore
I'm extracting from a Microsoft SQL database.
My preliminary tests have shown that I can't retrieve more than 3000 rows at
a time.
All this means that I would like to create a route that starts out by
extracting 3000 rows, at a time, as fast a possible. When all the 1.200.000
historic records have been extracted, the timer period should be changed to
only "fire" every 24 hours.
I have been looking into the throttler component, but I'm not sure that one
will work. The reason for this is that it will only be in a very short time,
that I want to poll as fast as possible, and after that I will only poll
every 24 hours for eternity. Furthermore, I can't figure out how the
throttler component know when to hold back. As I understand, the throttler
is controlled by how many exchanges that are going on right now, and
therefore only relates to the internal part of Camel. This means that the
throttler doesn't care about me only wanting data from a remote place every
24 hours.
Best regards
Lasse Vestergaard
--
View this message in context: http://camel.465427.n5.nabble.com/Change-timer-runtime-tp5738484.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Change timer runtime
Posted by Marco Westermann <Ma...@gmx.de>.
Hi,
I think you have more opinions to reach your goal.
One easy solution would be to use the quarz component instead of the
timer. You can configure the quarz component by a cron config. Therefor
you could configure that it should run as many times as you need to
process all rows.
http://camel.apache.org/quartz.html
Another solution would be to have a route with a timer and after all
rows have been processed you can stop the route and start it on the
next day at a certain time. To archive that have a look here:
http://camel.apache.org/how-can-i-stop-a-route-from-a-route.html
regards, Marco
Am 02.09.2013 10:16, schrieb lassesvestergaard:
> Hi all.
>
> I have a problem regarding changing the timer period on the fly. My concrete
> problem is that I have around 1.200.000 historic records I need to extract
> at one time, and the database table is updated every 24 hours. Furthermore
> I'm extracting from a Microsoft SQL database.
>
> My preliminary tests have shown that I can't retrieve more than 3000 rows at
> a time.
>
> All this means that I would like to create a route that starts out by
> extracting 3000 rows, at a time, as fast a possible. When all the 1.200.000
> historic records have been extracted, the timer period should be changed to
> only "fire" every 24 hours.
>
> I have been looking into the throttler component, but I'm not sure that one
> will work. The reason for this is that it will only be in a very short time,
> that I want to poll as fast as possible, and after that I will only poll
> every 24 hours for eternity. Furthermore, I can't figure out how the
> throttler component know when to hold back. As I understand, the throttler
> is controlled by how many exchanges that are going on right now, and
> therefore only relates to the internal part of Camel. This means that the
> throttler doesn't care about me only wanting data from a remote place every
> 24 hours.
>
> Best regards
>
> Lasse Vestergaard
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Change-timer-runtime-tp5738484.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Change timer runtime
Posted by lassesvestergaard <la...@gmail.com>.
Hi Claus.
Thanks for helping me out. I still don't get it, to be honest. This is my
code:
public class AffaldVarme extends RouteBuilder {
int latestID=1;
String resourceID="some_id";
@Override
public void configure() throws Exception {
from("timer://foo?period=1m")
.routeId("containervejning")
.setHeader("low",constant(latestID-1))
.setHeader("high",constant(latestID+3000))
.to("sql:select * from some_table where Container_Vejning_ID
> :#low and Container_Vejning_ID < :#high order by Container_Vejning_ID
ASC?dataSourceRef=affaldVarme")
//.setBody(constant("select * from some_table where Container_Vejning_ID
>"+(latestID-1)+" and Container_Vejning_ID <"+(latestID+3000)+" order by
Container_Vejning_ID ASC"))
.to("jdbc:affaldVarme")
.process(
new Processor() {
public void process(Exchange exchange) throws
Exception {
JSONArray ja=new JSONArray();
List<Map<String, Object>> data =
exchange.getIn().getBody(List.class);
for(Map<String, Object> row : data){
JSONObject jo=new JSONObject();
for(Map.Entry<String, Object> entry
:row.entrySet()){
jo.put(entry.getKey(),
entry.getValue());
}
ja.put(jo);
}
if(ja.length()>0){
JSONObject
j=(JSONObject)ja.get(ja.length()-1);
latestID=(Integer)j.get("Container_Vejning_ID");
}
exchange.getIn().setBody("{\"resource_id\":\""+resourceID+"\",\"fields\":["
+
"{\"id\":\"Container_Vejning_ID\",\"type\":\"int\"},"
+
"{\"id\":\"Indlaes_Vejning_ID\",\"type\":\"int\"},"
+ "{\"id\":\"Dato\",\"type\":\"text\"},"
+ "{\"id\":\"Tid\",\"type\":\"text\"},"
+
"{\"id\":\"Vejning\",\"type\":\"float\"},"
+
"{\"id\":\"Opd_Init\",\"type\":\"text\"},"
+
"{\"id\":\"Opd_Dato\",\"type\":\"text\"},"
+
"{\"id\":\"GPSLongitude_2\",\"type\":\"float\"},"
+
"{\"id\":\"GPSLatitude_2\",\"type\":\"float\"},"
+
"{\"id\":\"Container_Vejning_Faktura_Opstil_id\",\"type\":\"int\"},"
+
"{\"id\":\"FrivaegtKg\",\"type\":\"text\"}"
+ "] ,\"records\": " + ja.toString() +
"}");
}
}).setHeader("CamelHttpMethod", constant("POST"))
.setHeader("Authorization", constant("some_value"))
.to("http4://some_url");
}
}
As you can see, I have these two versions of calling the database:
.setHeader("low",constant(latestID-1))
.setHeader("high",constant(latestID+3000))
.to("sql:select * from some_table where Container_Vejning_ID > :#low and
Container_Vejning_ID < :#high order by Container_Vejning_ID
ASC?dataSourceRef=affaldVarme")
.setBody(constant("select * from some_table where Container_Vejning_ID
>"+(latestID-1)+" and Container_Vejning_ID <"+(latestID+3000)+" order by
Container_Vejning_ID ASC"))
.to("jdbc:affaldVarme")
Neither of these works (well... I can make the route active and run without
errors, but "latestID" will not increment past 3000), and to be hones, even
with the link you just posted, I can't make sense of it. As far as I can
see, I am using a different language, than constant, in my first example.
The thing is that both ways of doing the calls yields the same result.
Any suggestion?
Regards
Lasse Vestergaard
--
View this message in context: http://camel.465427.n5.nabble.com/Change-timer-runtime-tp5738484p5738610.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Change timer runtime
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Instead of constant language you can use any of the other langauges
that can be dynamic to construct your SQL query
http://camel.apache.org/languages
On Tue, Sep 3, 2013 at 4:59 PM, lassesvestergaard
<la...@gmail.com> wrote:
> Hi guys.
>
> Thanks for your replies. I will look further into the route policy area.
>
> In relation to this issue I have an other problem. I use the jdbc component
> to retrieve data from the Microsoft database. When I set the body, I do that
> by writing something like:
>
> setBody(constant("sql_sentence")). My problem is that I can't have dynamic
> sql queries. I guess it's because I use the constant expression. I'm using
> camel 2.11.
>
> Do you have any suggestions on how to have dynamic sql queries?
>
> Best regards
>
> Lasse Vestergaard
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Change-timer-runtime-tp5738484p5738596.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: Change timer runtime
Posted by lassesvestergaard <la...@gmail.com>.
Hi guys.
Thanks for your replies. I will look further into the route policy area.
In relation to this issue I have an other problem. I use the jdbc component
to retrieve data from the Microsoft database. When I set the body, I do that
by writing something like:
setBody(constant("sql_sentence")). My problem is that I can't have dynamic
sql queries. I guess it's because I use the constant expression. I'm using
camel 2.11.
Do you have any suggestions on how to have dynamic sql queries?
Best regards
Lasse Vestergaard
--
View this message in context: http://camel.465427.n5.nabble.com/Change-timer-runtime-tp5738484p5738596.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Change timer runtime
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
You can also use a route policy, and then just mark the exchange to
stop if you do not want to route anymore today. Then the timer keeps
running.
There is an onExchangeBegin method where you can mark the exchange to stop with
// mark the exchange to stop continue routing
exchange.setProperty(Exchange.ROUTE_STOP, Boolean.TRUE);
On Mon, Sep 2, 2013 at 10:16 AM, lassesvestergaard
<la...@gmail.com> wrote:
> Hi all.
>
> I have a problem regarding changing the timer period on the fly. My concrete
> problem is that I have around 1.200.000 historic records I need to extract
> at one time, and the database table is updated every 24 hours. Furthermore
> I'm extracting from a Microsoft SQL database.
>
> My preliminary tests have shown that I can't retrieve more than 3000 rows at
> a time.
>
> All this means that I would like to create a route that starts out by
> extracting 3000 rows, at a time, as fast a possible. When all the 1.200.000
> historic records have been extracted, the timer period should be changed to
> only "fire" every 24 hours.
>
> I have been looking into the throttler component, but I'm not sure that one
> will work. The reason for this is that it will only be in a very short time,
> that I want to poll as fast as possible, and after that I will only poll
> every 24 hours for eternity. Furthermore, I can't figure out how the
> throttler component know when to hold back. As I understand, the throttler
> is controlled by how many exchanges that are going on right now, and
> therefore only relates to the internal part of Camel. This means that the
> throttler doesn't care about me only wanting data from a remote place every
> 24 hours.
>
> Best regards
>
> Lasse Vestergaard
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Change-timer-runtime-tp5738484.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen