You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by Lindgren Peter <Pe...@volvofinans.se.INVALID> on 2023/01/10 11:22:08 UTC
camel-quartz URL decode
Hello! My first posting on this list, please be gentle 😊
We’ve been using Apache Camel for a short while, and now we have started to use Quartz as the scheduler for SFTP transfers. We have two nodes, and we've decided on the use of Quartz for clustering.
To simplify for our developers and integrators to set up new SFTP transfers (and migrate from other platform), we have an SFTP routebuilder that sets the common URL parameters that fits our environment and usage, to keep the noise/clutter down as much as possible in the properties files.
All parameters are added using the Camel's URISupport library, which URL-encodes the parameters. Thus, it also encodes the quartz-related parameters -- the cron string and time zone ID, which the QuartzScheduledPollConsumerScheduler can't decode (it merely replaces '+' with space in the cron string). It seems to be the same in all 3.x versions, and in master.
I did some experiments locally with 3.14.8 and came up with a few simple changes in QuartzScheduledPollConsumerScheduler which solves my problem, and I think it is a general solution.
How would you suggest I continue? Should I make a jira ticket before I push my changes in my fork? I guess I can't just create my own branch in the main repo.
Peter
Re: camel-quartz URL decode
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Have you seen org.apache.camel.spi.EndpointUriFactory which can be used to
build URIs from a set of Map parameters etc.
Can you look at that first before we consider doing any changes in current
code
On Wed, Jan 11, 2023 at 7:31 PM Lindgren Peter
<Pe...@volvofinans.se.invalid> wrote:
> Hi!
> Basically, this is what we're doing with the SFTP endpoint URL (we're
> adding several other SFTP parameters in the same way, not shown here for
> clarity):
>
>
> Map<String, Object> existingParameters = new CaseInsensitiveMap();
> String uriQueryPart = URISupport.extractQuery(urlWithParameters);
> existingParameters = URISupport.parseQuery(uriQueryPart, true); // useRaw
> = true
>
> CaseInsensitiveMap useParameters = new CaseInsensitiveMap();
> useParameters.put("scheduler", "quartz");
> useParameters.put("scheduler.cron", cron);
> useParameters.put("scheduler.triggerGroup", integrationId);
> useParameters.put("scheduler.triggerId", routeName);
> useParameters.put("scheduler.timeZone", "Europe/Stockholm");
> useParameters.putAll(existingParameters);
> String uriNoQuery = URISupport.stripQuery(urlWithParameters);
> return URISupport.appendParametersToURI(uriNoQuery, useParameters);
>
>
> The changes I propose in QuartzScheduledPollConsumerScheduler.java is in
> two places, both to perform URL decoding, which is safe for already-decoded
> strings:
>
> * Change+move the cron.replace from doStart() to
> public void setCron(String cron) {
> this.cron = URLDecoder.decode(cron, StandardCharsets.UTF_8);
> }
>
> * Adding a new setter for TimeZone with a String. This will cause the base
> code to use the String setter instead, but existing code that uses a
> TimeZone object will continue to work as before:
> public void setTimeZone(String timeZoneId) {
> this.timeZone = TimeZone.getTimeZone(URLDecoder.decode(timeZoneId,
> StandardCharsets.UTF_8));
> }
>
> This way, the quartz scheduler will be safe to use URL-encoded parameters
> in the same way as, for example, the SFTP component.
> It will introduce dependencies to java.net and java.nio:
>
> import java.net.URLDecoder;
> import java.nio.charset.StandardCharsets;
>
> Peter
>
> -----Ursprungligt meddelande-----
> Från: Claus Ibsen <cl...@gmail.com>
> Skickat: tisdag den 10 januari 2023 20:36
> Kopia: dev@camel.apache.org
> Ämne: [*Newsletter*] Re: camel-quartz URL decode
>
> Varning! Externt mail. Klicka inte på länkar eller bifogade filer i mail
> där avsändaren eller innehållet är okänt för dig.
>
> Hi
>
> Hmm how are you configuring that quartz endpoint exactly. As it seems you
> go some custom way with URISupport and encoding and whatnot.
> You can configure Camel endpoints as basic strings, or use the Endpoint
> DSL etc.
>
> We are not so keen on changing code in endpoint / URI without a compelling
> reason, especially in patch releases.
>
> So what we need is if you can put a sample project code somewhere that
> demonstrates this, and we can take a look at that first.
>
> On Tue, Jan 10, 2023 at 12:22 PM Lindgren Peter
> <Pe...@volvofinans.se.invalid> wrote:
>
> > Hello! My first posting on this list, please be gentle 😊
> > We’ve been using Apache Camel for a short while, and now we have
> > started to use Quartz as the scheduler for SFTP transfers. We have two
> > nodes, and we've decided on the use of Quartz for clustering.
> >
> > To simplify for our developers and integrators to set up new SFTP
> > transfers (and migrate from other platform), we have an SFTP
> > routebuilder that sets the common URL parameters that fits our
> > environment and usage, to keep the noise/clutter down as much as
> possible in the properties files.
> >
> > All parameters are added using the Camel's URISupport library, which
> > URL-encodes the parameters. Thus, it also encodes the quartz-related
> > parameters -- the cron string and time zone ID, which the
> > QuartzScheduledPollConsumerScheduler can't decode (it merely replaces '+'
> > with space in the cron string). It seems to be the same in all 3.x
> > versions, and in master.
> >
> > I did some experiments locally with 3.14.8 and came up with a few
> > simple changes in QuartzScheduledPollConsumerScheduler which solves my
> > problem, and I think it is a general solution.
> > How would you suggest I continue? Should I make a jira ticket before I
> > push my changes in my fork? I guess I can't just create my own branch
> > in the main repo.
> >
> > Peter
> >
>
>
> --
> Claus Ibsen
> -----------------
> @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>
--
Claus Ibsen
-----------------
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: camel-quartz URL decode
Posted by Lindgren Peter <Pe...@volvofinans.se.INVALID>.
Hi!
Basically, this is what we're doing with the SFTP endpoint URL (we're adding several other SFTP parameters in the same way, not shown here for clarity):
Map<String, Object> existingParameters = new CaseInsensitiveMap();
String uriQueryPart = URISupport.extractQuery(urlWithParameters);
existingParameters = URISupport.parseQuery(uriQueryPart, true); // useRaw = true
CaseInsensitiveMap useParameters = new CaseInsensitiveMap();
useParameters.put("scheduler", "quartz");
useParameters.put("scheduler.cron", cron);
useParameters.put("scheduler.triggerGroup", integrationId);
useParameters.put("scheduler.triggerId", routeName);
useParameters.put("scheduler.timeZone", "Europe/Stockholm");
useParameters.putAll(existingParameters);
String uriNoQuery = URISupport.stripQuery(urlWithParameters);
return URISupport.appendParametersToURI(uriNoQuery, useParameters);
The changes I propose in QuartzScheduledPollConsumerScheduler.java is in two places, both to perform URL decoding, which is safe for already-decoded strings:
* Change+move the cron.replace from doStart() to
public void setCron(String cron) {
this.cron = URLDecoder.decode(cron, StandardCharsets.UTF_8);
}
* Adding a new setter for TimeZone with a String. This will cause the base code to use the String setter instead, but existing code that uses a TimeZone object will continue to work as before:
public void setTimeZone(String timeZoneId) {
this.timeZone = TimeZone.getTimeZone(URLDecoder.decode(timeZoneId, StandardCharsets.UTF_8));
}
This way, the quartz scheduler will be safe to use URL-encoded parameters in the same way as, for example, the SFTP component.
It will introduce dependencies to java.net and java.nio:
import java.net.URLDecoder;
import java.nio.charset.StandardCharsets;
Peter
-----Ursprungligt meddelande-----
Från: Claus Ibsen <cl...@gmail.com>
Skickat: tisdag den 10 januari 2023 20:36
Kopia: dev@camel.apache.org
Ämne: [*Newsletter*] Re: camel-quartz URL decode
Varning! Externt mail. Klicka inte på länkar eller bifogade filer i mail där avsändaren eller innehållet är okänt för dig.
Hi
Hmm how are you configuring that quartz endpoint exactly. As it seems you go some custom way with URISupport and encoding and whatnot.
You can configure Camel endpoints as basic strings, or use the Endpoint DSL etc.
We are not so keen on changing code in endpoint / URI without a compelling reason, especially in patch releases.
So what we need is if you can put a sample project code somewhere that demonstrates this, and we can take a look at that first.
On Tue, Jan 10, 2023 at 12:22 PM Lindgren Peter <Pe...@volvofinans.se.invalid> wrote:
> Hello! My first posting on this list, please be gentle 😊
> We’ve been using Apache Camel for a short while, and now we have
> started to use Quartz as the scheduler for SFTP transfers. We have two
> nodes, and we've decided on the use of Quartz for clustering.
>
> To simplify for our developers and integrators to set up new SFTP
> transfers (and migrate from other platform), we have an SFTP
> routebuilder that sets the common URL parameters that fits our
> environment and usage, to keep the noise/clutter down as much as possible in the properties files.
>
> All parameters are added using the Camel's URISupport library, which
> URL-encodes the parameters. Thus, it also encodes the quartz-related
> parameters -- the cron string and time zone ID, which the
> QuartzScheduledPollConsumerScheduler can't decode (it merely replaces '+'
> with space in the cron string). It seems to be the same in all 3.x
> versions, and in master.
>
> I did some experiments locally with 3.14.8 and came up with a few
> simple changes in QuartzScheduledPollConsumerScheduler which solves my
> problem, and I think it is a general solution.
> How would you suggest I continue? Should I make a jira ticket before I
> push my changes in my fork? I guess I can't just create my own branch
> in the main repo.
>
> Peter
>
--
Claus Ibsen
-----------------
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: camel-quartz URL decode
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Hmm how are you configuring that quartz endpoint exactly. As it seems you
go some custom way with URISupport and encoding and whatnot.
You can configure Camel endpoints as basic strings, or use the Endpoint DSL
etc.
We are not so keen on changing code in endpoint / URI without a
compelling reason, especially in patch releases.
So what we need is if you can put a sample project code somewhere that
demonstrates this, and we can take a look at that first.
On Tue, Jan 10, 2023 at 12:22 PM Lindgren Peter
<Pe...@volvofinans.se.invalid> wrote:
> Hello! My first posting on this list, please be gentle 😊
> We’ve been using Apache Camel for a short while, and now we have started
> to use Quartz as the scheduler for SFTP transfers. We have two nodes, and
> we've decided on the use of Quartz for clustering.
>
> To simplify for our developers and integrators to set up new SFTP
> transfers (and migrate from other platform), we have an SFTP routebuilder
> that sets the common URL parameters that fits our environment and usage, to
> keep the noise/clutter down as much as possible in the properties files.
>
> All parameters are added using the Camel's URISupport library, which
> URL-encodes the parameters. Thus, it also encodes the quartz-related
> parameters -- the cron string and time zone ID, which the
> QuartzScheduledPollConsumerScheduler can't decode (it merely replaces '+'
> with space in the cron string). It seems to be the same in all 3.x
> versions, and in master.
>
> I did some experiments locally with 3.14.8 and came up with a few simple
> changes in QuartzScheduledPollConsumerScheduler which solves my problem,
> and I think it is a general solution.
> How would you suggest I continue? Should I make a jira ticket before I
> push my changes in my fork? I guess I can't just create my own branch in
> the main repo.
>
> Peter
>
--
Claus Ibsen
-----------------
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2