You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ant.apache.org by Kev Jackson <ke...@it.fts-vn.com> on 2006/05/09 12:10:42 UTC
Scp is this possible?
Hi all,
I'm messing with a hideous integration project and part of the
requirements involve aggregating log files from servers via a web
interface. I'm using ant's scp task to try and grab the log files (as
we are not allowed to install anything on the servers).
What I want to do is to use a fileset to filter the log files *before*
pulling them back to the aggregation server. I thought that this was
possible as scp takes a fileset which can use different selectors
ie
<scp localToDir="${basedir}" trust="true">
<fileset
dir="weblogic:weblogic@172.16.1.15:/home/weblogic/logs" includes="*.log" />
</scp>
Ok this is obviously not a realistic example as I can simply use a
wildcard in the file attribute to achive this, but my real use case
involves filtering the logs based on from/to dates - I thought the new
TimeComparison stuff would be perfect for this, but I cannot get scp to
work with todir set to a normal path, it always wants
user:password@host:/path, which isn't what I want.
So basically two questions:
1 - is it possible to use scp task with a fileset in this manner (ie
filter files on remote server before pulling back)
2 - any work arounds if it's impossible
Thanks
Kev
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: Scp is this possible?
Posted by Stefan Bodewig <bo...@apache.org>.
On Wed, 10 May 2006, Kev Jackson <ke...@it.fts-vn.com> wrote:
> Would it be possible to alter DateSelector to use resources
It does, if you use the right one
(org.apache.tools.ant.taskdefs.resources.selectors.Date) ;-)
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: Scp is this possible?
Posted by Kev Jackson <ke...@it.fts-vn.com>.
Atsuhiko Yamanaka wrote:
> Hi,
>
> 2006/5/10, Kev Jackson <ke...@it.fts-vn.com>:
>
>> In the SFTP code, there's mention of selectors not supported on remote
>> filesystems. Why is taht? The SFTPDirectoryScanner inherits from the
>> normal DirectoyScanner (which does support selectors), is there a
>> problem with the SFTP protocol that will not allow selectors to work?
>
>
> That logic comes from ftp task, but I guess that DateSelector expects
> only
> java.io.File instances, but remote files are not java.io.File.
Would it be possible to alter DateSelector to use resources instead and
see if that works?
> Anyway, after quick and ugly hack, I could get files with date selector.
>
At least you got that far, to be honest I've not managed to look through
all the code and determine what I would need to do to implement any of
the selectors
> In that hack, I get an instance of DateSelector and invoke its
> 'isSelected' method
> whith dummy java.io.File instance whose 'lastModifiedTime' is setted
> as timestam of
> remote file. It is too ugly to post here, so I'll wait for reponses
> from Ant experts.
Currently I'm hacking around with building a list of include patterns
based on filenames (each of the log files will have a filename
corresponding to the date pattern YYYY-MM-DD) - this is so brittle it's
untrue, so I'd be happier to use the 'hacked up' code you've put
together for my use case - I doubt anyone would ever need to do this
again, so I'm not bothered about how messy the code is in the jar files
(I'm treating Ant as a simple lib), and I doubt the rest of the ant team
would want it to be included in the distribution
...horrible kludgy code to prove the hoops I'm having to jump through
ArrayList includes = new ArrayList();
try {
SimpleDateFormat sdf = new SimpleDateFormat("YYYY-MM-DD");
Date now = new Date();
now = sdf.parse((sdf.format(now)));
if (null != from && null == to) {
Date f = sdf.parse(from);
System.out.println("Attempting to select files after :
"+from);
//after chosen date
//ds.setDatetime(from); <- elegant using date selector
//ds.setWhen(TimeComparison.AFTER);
//fs.addDate(ds);
Calendar c = Calendar.getInstance(); <- nasty using
clandar etc
//loop around calendar adding one to each day
//until calendar == today
for (c.setTime(f); c.getTime().equals(now);
c.add(Calendar.DAY_OF_MONTH, 1)) {
includes.add("*"+sdf.format(c.getTime())+"*");
System.out.println("*"+sdf.format(c.getTime())+"*");
}
.... code elided
} catch (ParseException e) {
//do nothing
}
String[] includePatterns = new String[includes.size()];
for(int i = 0, size = includes.size(); i<size; i++) {
includePatterns[i] = (String)includes.get(i);
}
fs.appendIncludes(includePatterns);
return fs;
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: Scp is this possible?
Posted by Atsuhiko Yamanaka <ym...@jcraft.com>.
Hi,
2006/5/10, Kev Jackson <ke...@it.fts-vn.com>:
> In the SFTP code, there's mention of selectors not supported on remote
> filesystems. Why is taht? The SFTPDirectoryScanner inherits from the
> normal DirectoyScanner (which does support selectors), is there a
> problem with the SFTP protocol that will not allow selectors to work?
That logic comes from ftp task, but I guess that DateSelector expects only
java.io.File instances, but remote files are not java.io.File.
Anyway, after quick and ugly hack, I could get files with date selector.
In that hack, I get an instance of DateSelector and invoke its
'isSelected' method
whith dummy java.io.File instance whose 'lastModifiedTime' is setted
as timestam of
remote file. It is too ugly to post here, so I'll wait for reponses
from Ant experts.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: Scp is this possible?
Posted by Kev Jackson <ke...@it.fts-vn.com>.
> By itself this works fine, the problem is that i need
>
> <fileset dir="${basedir}">
> <date datetime="2006-05-10" when="after" pattern="YYYY-MM-DD"/>
> </fileset>
>
In the SFTP code, there's mention of selectors not supported on remote
filesystems. Why is taht? The SFTPDirectoryScanner inherits from the
normal DirectoyScanner (which does support selectors), is there a
problem with the SFTP protocol that will not allow selectors to work?
Thanks
Kev
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: Scp is this possible?
Posted by Kev Jackson <ke...@it.fts-vn.com>.
Atsuhiko Yamanaka wrote:
> Hi,
>
> 2006/5/10, Kev Jackson <ke...@it.fts-vn.com>:
>
>> Damnit!
>>
>> "selectors are not supported on remote filesets"
>>
>> Basically that's what I need, the ability to select the files based on
>> the last modified - I don't want to pull back files if they are not
>> within the search criteria :(
>
>
> By following task, I can get only newer files matched with "*.log" on
> the remote 'home/weblogic/logs' and save to ${basedir}.
>
> <sftp action="get"
> trust="true"
> server="${host}"
> userid="${user}"
> password="${password}"
> depends="yes"
> remotedir="/home/weblogic/logs">
> <fileset dir="${basedir}" >
> <include name="*.log"/>
> </fileset>
> </sftp>
>
> # Please note that 'depends="yes"'.
>
> Does it not work on your environment or is it enougth?
By itself this works fine, the problem is that i need
<fileset dir="${basedir}">
<date datetime="2006-05-10" when="after" pattern="YYYY-MM-DD"/>
</fileset>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: Scp is this possible?
Posted by Atsuhiko Yamanaka <ym...@jcraft.com>.
Hi,
2006/5/10, Kev Jackson <ke...@it.fts-vn.com>:
> Damnit!
>
> "selectors are not supported on remote filesets"
>
> Basically that's what I need, the ability to select the files based on
> the last modified - I don't want to pull back files if they are not
> within the search criteria :(
By following task, I can get only newer files matched with "*.log" on
the remote 'home/weblogic/logs' and save to ${basedir}.
<sftp action="get"
trust="true"
server="${host}"
userid="${user}"
password="${password}"
depends="yes"
remotedir="/home/weblogic/logs">
<fileset dir="${basedir}" >
<include name="*.log"/>
</fileset>
</sftp>
# Please note that 'depends="yes"'.
Does it not work on your environment or is it enougth?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: Scp is this possible?
Posted by Kev Jackson <ke...@it.fts-vn.com>.
Kev Jackson wrote:
> Atsuhiko Yamanaka wrote:
>
>> Hi,
>>
>> 2006/5/10, Stefan Bodewig <bo...@apache.org>:
>>
>>> On Wed, 10 May 2006, Kevin Jackson <fo...@gmail.com> wrote:
>>>
>>> > Thanks, I will certainly try it and use it if it works.
>>>
>>> If you want to apply ymnk's patch, I committed everything except fpr
>>> the files starting with all uppercase SCP.
>>
>>
>>
>> FYI, that attached patch[1] on Bug entry 39532[2] includes only files
>> for sftp task
>> and it must be applied to ant_20060509104815.tar.gz without any errors.
>
>
> I've applied the patch to my local trunk of ant, I made a couple of
> minor changes (mainly getting rid of unused imports etc) - so far so
> good, I can get the files from the remote machine using a contains
> selector, next test is the difficult one, with two date selectors to
> implement between. (ie from-date < file < to-date)
Damnit!
"selectors are not supported on remote filesets"
Basically that's what I need, the ability to select the files based on
the last modified - I don't want to pull back files if they are not
within the search criteria :(
Atsuhiko Yamanaka:
Is this impossible to implement, or did you not need it and therefore
cut it out from the patch?
Thanks
Kev
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: Scp is this possible?
Posted by Stefan Bodewig <bo...@apache.org>.
On Wed, 10 May 2006, Kev Jackson <ke...@it.fts-vn.com> wrote:
> For all SSH related stuff, is the consensus to keep it as an
> optional task, or would an antlib make sense for 1.7?
This is a question for a different thread.
Personally I'd be in favor of moving all stuff that requires a third
party library out of the main tree and into antlibs but am unsure we
should takle that for 1.7.0.
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: Scp is this possible?
Posted by Kev Jackson <ke...@it.fts-vn.com>.
Atsuhiko Yamanaka wrote:
> Hi,
>
> 2006/5/10, Stefan Bodewig <bo...@apache.org>:
>
>> On Wed, 10 May 2006, Kevin Jackson <fo...@gmail.com> wrote:
>>
>> > Thanks, I will certainly try it and use it if it works.
>>
>> If you want to apply ymnk's patch, I committed everything except fpr
>> the files starting with all uppercase SCP.
>
>
> FYI, that attached patch[1] on Bug entry 39532[2] includes only files
> for sftp task
> and it must be applied to ant_20060509104815.tar.gz without any errors.
I've applied the patch to my local trunk of ant, I made a couple of
minor changes (mainly getting rid of unused imports etc) - so far so
good, I can get the files from the remote machine using a contains
selector, next test is the difficult one, with two date selectors to
implement between. (ie from-date < file < to-date)
Fantastic timing of the patch by the way, makes my life about 1000%
simpler :)
For all SSH related stuff, is the consensus to keep it as an optional
task, or would an antlib make sense for 1.7?
Kev
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: Scp is this possible?
Posted by Atsuhiko Yamanaka <ym...@jcraft.com>.
Hi,
2006/5/10, Stefan Bodewig <bo...@apache.org>:
> On Wed, 10 May 2006, Kevin Jackson <fo...@gmail.com> wrote:
>
> > Thanks, I will certainly try it and use it if it works.
>
> If you want to apply ymnk's patch, I committed everything except fpr
> the files starting with all uppercase SCP.
FYI, that attached patch[1] on Bug entry 39532[2] includes only files
for sftp task
and it must be applied to ant_20060509104815.tar.gz without any errors.
[1] http://issues.apache.org/bugzilla/attachment.cgi?id=18250
[2] http://issues.apache.org/bugzilla/show_bug.cgi?id=39532
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: Scp is this possible?
Posted by Stefan Bodewig <bo...@apache.org>.
On Wed, 10 May 2006, Kevin Jackson <fo...@gmail.com> wrote:
> Thanks, I will certainly try it and use it if it works.
If you want to apply ymnk's patch, I committed everything except fpr
the files starting with all uppercase SCP.
Please let us know what you find.
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: Scp is this possible?
Posted by Kevin Jackson <fo...@gmail.com>.
> I have posted a patch[1] to implement sftp task.
Yes it was discussed yesterday on the dev list
> By using sftp task, you can do as follows,
>
> <sftp action="get"
> trust="true"
> server="172.16.1.15"
> userid="weblogic"
> password="weblogic"
> depends="yes"
> remotedir="/home/weblogic/logs">
> <fileset dir="${basedir}" >
> <include name="*.log"/>
> </fileset>
> </sftp>
>
Assuming that the fileset operates on the remote side, that is exactly
what I want - I will give it a try yes.
> If you are interested in it, please try it.
>
Thanks, I will certainly try it and use it if it works.
> [1] http://issues.apache.org/bugzilla/show_bug.cgi?id=39532
>
Kev
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: Scp is this possible?
Posted by Atsuhiko Yamanaka <ym...@jcraft.com>.
Hi,
2006/5/9, Kev Jackson <ke...@it.fts-vn.com>:
> 2 - any work arounds if it's impossible
I have posted a patch[1] to implement sftp task.
By using sftp task, you can do as follows,
<sftp action="get"
trust="true"
server="172.16.1.15"
userid="weblogic"
password="weblogic"
depends="yes"
remotedir="/home/weblogic/logs">
<fileset dir="${basedir}" >
<include name="*.log"/>
</fileset>
</sftp>
If you are interested in it, please try it.
[1] http://issues.apache.org/bugzilla/show_bug.cgi?id=39532
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: Scp is this possible?
Posted by Kevin Jackson <fo...@gmail.com>.
>
> or jsch
>
> http://www.jcraft.com/jsch/
>
I'm already using this one (wrapped with the ant task). I decided to
use the Ant task for two reasons
1) I can test certain things just using ant without having to deploy a
full web application
2) The SSHExec and Scp tasks are well tested and well documented, why
re-invent the wheel
As for rsync, I agree it makes sense, but the customer will not allow
it on the servers.
Thanks for the suggestion
Kev
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: Scp is this possible?
Posted by Martin Menzel Apache <ma...@gmx.de>.
Hi Kevin,
with the ANT task I don't know how to solve your problem, but I have
used Mindterms SSH library
http://www.appgate.com/products/80_MindTerm/
or jsch
http://www.jcraft.com/jsch/
and you can execute ssh or scp command from within your java
application. If you think about to use rsync have a look to the jarsync
library.
Good luck
Martin
Kevin Jackson schrieb:
>>
>> ah the joy of a world where the ops team doesnt trust the developers.
>> Which is why I'm hosting my current endpoint on a laptop at home -I am
>> the ops team.
>>
>
> In this case it's more like, the joys of being told to do the
> impossible (use http to transfer log files from servers running on the
> network as it's the only 'sanctioned' protocol, unfortunately you
> aren't allowed to install anything that can respond to a http get/post
> request on said servers... (can you see where this is going yet)
>
> ssh is the compromise when they realised that what they wanted first
> time around was impossible
>
>> > 1 - is it possible to use scp task with a fileset in this manner (ie
>> > filter files on remote server before pulling back)
>>
>> Not according to the docs; filesets are for local stuff only.
>>
>> Maybe its time to add more features to scp.
>
> If it's not currently possible, I suppose that's one way to achieve it
>
>>
>> > 2 - any work arounds if it's impossible
>>
>> rsync.
>
> [sigh] I wish they'd allow me some freedom to build this when they
> want it next week.
>
> Kev
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
> For additional commands, e-mail: dev-help@ant.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: Scp is this possible?
Posted by Kevin Jackson <fo...@gmail.com>.
>
> ah the joy of a world where the ops team doesnt trust the developers.
> Which is why I'm hosting my current endpoint on a laptop at home -I am
> the ops team.
>
In this case it's more like, the joys of being told to do the
impossible (use http to transfer log files from servers running on the
network as it's the only 'sanctioned' protocol, unfortunately you
aren't allowed to install anything that can respond to a http get/post
request on said servers... (can you see where this is going yet)
ssh is the compromise when they realised that what they wanted first
time around was impossible
> > 1 - is it possible to use scp task with a fileset in this manner (ie
> > filter files on remote server before pulling back)
>
> Not according to the docs; filesets are for local stuff only.
>
> Maybe its time to add more features to scp.
If it's not currently possible, I suppose that's one way to achieve it
>
> > 2 - any work arounds if it's impossible
>
> rsync.
[sigh] I wish they'd allow me some freedom to build this when they
want it next week.
Kev
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: Scp is this possible?
Posted by Steve Loughran <st...@apache.org>.
Kev Jackson wrote:
> Hi all,
>
> I'm messing with a hideous integration project and part of the
> requirements involve aggregating log files from servers via a web
> interface. I'm using ant's scp task to try and grab the log files (as
> we are not allowed to install anything on the servers).
>
ah the joy of a world where the ops team doesnt trust the developers.
Which is why I'm hosting my current endpoint on a laptop at home -I am
the ops team.
> What I want to do is to use a fileset to filter the log files *before*
> pulling them back to the aggregation server. I thought that this was
> possible as scp takes a fileset which can use different selectors
>
> ie
>
> <scp localToDir="${basedir}" trust="true">
> <fileset
> dir="weblogic:weblogic@172.16.1.15:/home/weblogic/logs" includes="*.log" />
> </scp>
>
> Ok this is obviously not a realistic example as I can simply use a
> wildcard in the file attribute to achive this, but my real use case
> involves filtering the logs based on from/to dates - I thought the new
> TimeComparison stuff would be perfect for this, but I cannot get scp to
> work with todir set to a normal path, it always wants
> user:password@host:/path, which isn't what I want.
>
> So basically two questions:
>
> 1 - is it possible to use scp task with a fileset in this manner (ie
> filter files on remote server before pulling back)
Not according to the docs; filesets are for local stuff only.
Maybe its time to add more features to scp.
> 2 - any work arounds if it's impossible
rsync.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org