Can VFS SFTP be used to download todays files only

[kimhorn <ki...@icsglobal.net>]

We need to download files created only today on a SFTP server. Initially I
thought the regular expression parameter could be used to do this, but now I
don't think its possible.

A further major design fault of the VFS mediator is that I don't think these
parameters can be programatically set. 

Requirement: The SFTP server we need to connect to keeps every file the
partner creates, on it for months. We only need files with todays date in
the name. Unfortunately we cannot change the partners policy here or hwo
they name files.

The file naming scheme is:
CODEMMDDYYNN.TYPE
Where CODE is a set parameter
Where TYPE is a set parameter
Where NN is the increment number for files created within the day. So a *
here.
The rest is the date.

So the server may have 2000 files and 3 created today.
How can I download just those 3 ?

Originally I thought I could create the regular expresssion parameter in
Java code and pass to VFS.
But I don't think this is possible ?
VFS parameter are isolated from Synapse. Never thought Synapse would be so
inflexible.

Only solution left is to do it completely in a regular expression, not sure
this is feasible ?
Can a Java regular expression know the date today ?
Is this compatible with how VFS uses the expression ?

Is there any other way to set the VFS parameters ?

Alternative is to fix the way VFS parameters are set, e.g via a property
rather than inflexible text only. (However, if VFS is a proxy this cannot be
done either, because parameter cannot get set prior to proxy call)
OR
call VFS from Java - but then why use Synapse ?

A final solution is to use a Java program to write the synapse.xml file,
with the required text. Then hot-load the synapse.xml every midnight.
However, again, may as well just stick with Java solution.

This is about the last nail in the coffin for our problems with getting
Synapse to solve a simple mediation problem.






-- 
View this message in context: http://www.nabble.com/Can-VFS-SFTP-be-used-to-download-todays-files-only-tp22901452p22901452.html
Sent from the Synapse - User mailing list archive at Nabble.com.

RE: Can VFS SFTP be used to download todays files only

[Kim Horn <ki...@icsglobal.net>]

Another issue relating to VFS is that under windows, sometimes a script will not start polling or just stop. We use windows for development and Linux for production. As a result we may have to use our monitor to check the logs to see if VFS is always polling. I assume this is a windows issue at the moment so not too worried. Don't have conclusive results yet but wondering if anyone else has seen that ?

Thanks
Kim


-----Original Message-----
From: Kim Horn [mailto:kim.horn@icsglobal.net]
Sent: Thu 09/04/2009 08:00
To: user@synapse.apache.org
Subject: RE: Can VFS SFTP be used to download todays files only
 
I should note that we looked at using interval JMS queue and data base for resolving (5) below. An internal queue is a common soluion in Aqua Logic to do reliable messaging, when end point is down. The current solution uses local file system which is a less technically complex solution. The moves are not done but task completed status is stored in a DBMS, when transaction completes. We have to log this anyhow so was no real overhead.

Years ago we built an ESB for internal use in a large corporation, before ESBs were available as software. In that ESB we made reliable messaging a default function of the bus. It seems to be the base use case for any ESB, you always want a message to be guaranteed to be delivered. So no matter where in the ESB "script" the system failed it would start again in that same spot. If an end point was not available it would deliver later, given a certain number of re-tries.
We found the overhead of this, when it was made as a prime feature of the bus, was negligable.

Thanks
Kim


-----Original Message-----
From: Kim Horn [mailto:kim.horn@icsglobal.net]
Sent: Wed 08/04/2009 12:39
To: user@synapse.apache.org
Subject: RE: Can VFS SFTP be used to download todays files only
 
Hello Andreas,

Thanks for your concern. Yes I will do that, We have our own Jira for the project; and I will email the issues and a summary of the project as soon as we get over this hump. I am happy that we have not had to rule out Synapse totally and a hybrid approach may even be better; not sure ? 

I do have a list of Synapse Jira items, growing, that I will add, but I apologise for running out of time at the moment. When we get our current project up and running, I have allocated a week to work on Synapse stuff for you guys. 

In short and in relation to SFTP only the issues are:

1) we cannot have passwords in clear text in synapse scripts or anywhere. Due to the strict compliance we require; we can be seen as breaking US law doing this; one is the HIPPA Act. There are other compliance and accreditation standards we must comply too; that have very strict security policies. We review projects here so that we don't go to gaol (Jail in US) :-)
2) Ability to pass to VFS, parameters in variables (properties); so being able to download a file based on say a property value containing a name constructed, for example using a script or javainclude date.now().
3) I do not know if comments below are true but checking the 30 mb file limit; It seems this is a problem with Commons VFS SFTP and not Jsch. I am not sure what combination is used in Synapse VFS mediator. I know this has been addressed and I have tested as OK for VFS files.
4) upgrade to latest Jsch jar.
5) This one is a contentious issue: Roll back a VFS MOVE or DELETE, when the subsequent web service fails. This is like wrapping the SFTP get and WEB service put in a transaction. The issue we have now is that if a web service fails, the SFTP actions have been done as if successful. So the move has to be "compensated" by moving them back to the normal directory. Not sure if WS-Atomic transactions works in Synapse for SFTP; but that's the idea ? Not sure if transactions have been considered in general for Synapse ? I am not sure that this can be done using a failure sequence as just the move, without the download, on the SFTP server, is not possible ?

Of these only 1 to 3 impact us now, maybe 4.

Re 2: we are not happy that some partners do not allow us to move or delete files, but that's their policy and so what happens is that the files grow on their server and we currently use Date Modified to download latest. They also include the date of file in the file name. So we can use the Jsch lsat property getMTime() or build a file name regex using todays date to download new files. We currently have our system running in Australia and in the UK, we expanded to US and have found that SFTP is the most used B2B mechanism over there. We were Amazed. Basically in Australia Web Services are standard and no one uses SFTP, its dead. So this was the major attraction of Synapse to us.

It seems to me the base Use case for SFTP is to download a named file in text. Next is a regex. The next specialisation is to allow that name to be constructed programmatically. Next is to allow a regex to be constructed programmatically. More specialised use cases are the ones addressed by VFS mediator now where you use Move and Delete to get only new files only. For us it is rare to get this permission on other people servers. We do have one partner that is smart enough to do the move automatically, on their side, when we get a file. Other partners either allow us to delete or move files. So we have every combination.

Changing how VFS works now does not seem to be as easy, I guess as:
-  VFS Mediator may not be designed to be updated on the fly. I assume it would be only initialised on start up. The rqmt above requires the file regex to be changed each day, for the new day, or to be totally dynamic. Java regex, does not do date now; but it can be expanded to do this. I have some code for this.
-  It is not clear how you bootstrap this; as the property has to be set before a VFS proxy starts; hence why I avoided this solution now. This was my first experiment to try and reset the property for the regex after the first time the proxy executed Using:

<definitions xmlns="http://ws.apache.org/ns/synapse">
  <proxy name="StockQuoteProxy" transports="vfs">
	  <parameter name="transport.vfs.FileURI">file:///c:/test/in</parameter>
    <parameter name="transport.vfs.ContentType">text/plain</parameter>
    <parameter name="transport.vfs.FileNamePattern">.*\.edi</parameter>
    <parameter name="transport.PollInterval">15</parameter>
      <inSequence>
        <log level="full"/>
	<property name="transport.vfs.FileNamePattern" value=".*\.txt"/>
      </inSequence>
    </target>
  </proxy>
</definitions>

But this doesn't change the Pattern; only the ".edi" files get done :-)


Thanks all the best
Kim





-----Original Message-----
From: Andreas Veithen [mailto:andreas.veithen@gmail.com] 
Sent: Wednesday, 8 April 2009 7:13 AM
To: user@synapse.apache.org
Subject: Re: Can VFS SFTP be used to download todays files only

Kim,

Maybe at this point it would be useful if you could give a quick
overview of the open issues that you still face.

Andreas

On Tue, Apr 7, 2009 at 08:55, Kim Horn <ki...@icsglobal.net> wrote:
> Created a work around to this by writing an independent service using
> just Jsch & Quartz doing an sftp every hour.
> From Java we can pass in any expression we like and actually using Jsch
> can find the Latest Modified files from the system date. This also
> solves the SFTP passwords in clear text issue as now I can encrypt them
> in Java. So this service runs totally independent of Synapse. As a bonus
> this service can unzip SFTped zipped files we get from some locations
> and split large files as Jsch can tell us how big they are before
> SFTPing. A prior issue was submitting large files to some web services
> that required a new type of iterator in Synapse to be developed.
>
> So now Synapse just uses VFS to poles the file system, the service
> downloads files too, to submit manageable size text files to the
> destination Web Services.
>
> From the other Forums this solution also will resolve a possible issue
> with Commons VFS SFTP having problems with large files. Not sure if
> these issues are real but dates are fairly recent.
>
> Start Quotes:
>
> "This doesn't work with larger files. The VFS seems to have an issue
> with the getInputStream(), where it loads the whole file into memory
> before returning a memory stream. Eventually the JVM will run out of
> memory. In any case, it was a nice start--I'm going to try using Jsch to
> work around this problem.
> # posted by Blogger Wes : 11:34 PM, November 24, 2008
>
> Yes. The code is not working with larger files. It's OK up to 30mb
> files. If it exceeds more than 30mb it's causing problems(memory
> issues). Any idea how to get rid of this problem.
> # posted by Anonymous Anonymous : 3:17 PM, December 16, 2008.
>
> End Quotes.
>
>
> I have used a more recent release of Jsch than the Jar used in Synapse;
> that requires a later version of Java, so would not be possible to do
> this as a Synapse class mediator at the moment, although this would be a
> better solution.
>
> Thanks
> Kim
>
>
>
> -----Original Message-----
> From: Kim Horn [mailto:kim.horn@icsglobal.net]
> Sent: Tuesday, 7 April 2009 7:19 AM
> To: user@synapse.apache.org
> Subject: RE: Can VFS SFTP be used to download todays files only
>
> Hello Asankha,
>
> yes we did get some small simple scripts into production, to help our
> manual process, in parralel with evaluation, but had to pull them out as
> they did not run with the latest trunk required to solve another issue;
> the VFS memory not being released. The fix to this resulted in the "out"
> sequences producing exceptions with VFS.
>
> Thanks
> Kim
>
>
>
> -----Original Message-----
> From: Asankha Perera on behalf of Asankha C. Perera
> Sent: Mon 06/04/2009 20:20
> To: user@synapse.apache.org
> Subject: Re: Can VFS SFTP be used to download todays files only
>
> Hi Kim
>> 1) How do you specify a regex to download files with todays date that
> will
>> run in synapse unattended for weeks ?
>> Yes I can hard code an expression to match 31 March 2009 and after but
> that
>> is no good today, is it.
>> Java has the concept of Date NOW; this is what I mean by today.
>>
>> You say this is "Quite easy" please show me how, as I cannot see how
> this
>> can be done ? If it is easy then my problem is solved. Great, but this
> is my
>> question ?
>>
>> 2) �If an SFTP server has 2000 old files permanently on it now that
> will not
>> be removed; thats grows every day. How to I download just the new
> files ?
>> E.G Those placed on there with todays date in the name. I have no
> permission
>> on the partners SFTP server to MOVE files or DELETE them, they just
> stay
>> there. It is not a good assumption that files can be MOVED or DELETED
> on
>> another organisations SFTP server. Very rarely in real B2B do you get
> this
>> luxury.
>>
> The real world cases for which the transport has been originally written
>
> is to poll for a file or files at some specific location that matches
> some pattern, process them, and either 1) delete or 2) move them since
> the transport does not have state to remember what has been polled
> already.
>
> The commercial Aqualogic service bus follows this pattern too [1], while
>
> both Mule and Camel additionally supports the specification of date info
>
> in the file pattern. I couldn't readily find much about Fuse or JBoss
> etc.
>
> The solution I gave will not work, if you cannot move or delete
> processed files on the target sftp server, to prevent re-processing them
>
> later. However, supporting of date information in the pattern would be
> fairly easy to implement.
>> 3) You suggest SYnapse has a concept of new files ? How does this work
> ?
>>
> This is if you are creating a new file.. e.g. saving a message received
> via http/s, JMS, email etc as a file - maybe to an sftp location like in
> [2]
>> 4) Are you saying that when I start Synapse with an SFTP mediator it
>> remembers every file downloaded ever in the past; so that when we
> re-start
>> Synapse (after maintenence) it remembers the old files. If this is
> true this
>> is great. I am interested to know where this record of files is kept.
> Where
>> is the file ?
>>
>> 5) Otherwise I am missing some very interesting behaviour of the
> Synapse VFS
>> Mediator that does not seem to be provided by the samples or the
>> documentation or any prior Forum discussions.
>>
> No, the transport is implemented as whats called an "Axis2 transport" ..
>
> its actually out of the Synapse core, and not directly dependent or
> linked. Implementing a record of processed files - even if possible
> would not be worth it - I think the use of Date in the regex is the most
>
> simple way to handle this
>> I am quite happy to raise a Jira but there is no need if Synapse can
> do
>> this, easily, as you say ?
>>
> Well.. the point of a JIRA is that all good ideas are captured by those
> who propose them, so that new developers or the current developers -
> when they have free time, can look at real world user suggestions for
> improvement and implement them for future versions
>> I do not understand this comment about being in a hurry; we are
> evaluating
>> Synapse to solve a commercial problem and unfortunately we have
> encountered
>> many issues that we are trying to resolve.
> I interpreted your multiple emails in the past about dropping or almost
> dropping the use of Synapse to mean that you are in a hurry to get this
> implemented in Synapse, and thats why I even suggested a pure Java
> implementation. Of course if you are still in evaluation, the best is to
>
> try to select another few ESBs (open source and/or commercial) and try
> to implement the same, and consider the total cost and the complexity of
>
> the final solution. In this case, consider the effort to implement whats
>
> missing in Synapse into the total cost of the solution.
>
> However, from some of your emails, I see production requirements being
> talked about, and thats why sometime back I suggested getting commercial
>
> support - in which case, someone else would have taken care of all of
> these issues - implementing what was missing as required. To be honest,
> you could have got a complete solution for your real world problem, with
>
> commercial support and a guarantee, probably for much less than the
> number of hours you've already spent and would spend * a reasonable
> hourly rate for the cost of your time, which again is money.
>
>> Unfortunately when I think all the hurdles are solved; then we stumble
> over new ones.
>> Yes unfortunately real world projects have definite requirements and
> budgets and deadlines.
>> I do not see the point of open source software if it cannot solve
> commercial
>> problems; unless Synapse is just an Academic Research project ? I am
> sure
>> WS02 are using Synapse to solve real world commercial problems and not
> just
>> for research. I am sure they make money from using the code.
>>
> Of course, many real world users use Synapse - but some are shy to use
> the public forums, and risk exposing confidential information, security
> and passwords and sensitive data etc. I usually privately inform users
> whenever this happens.. since logs could reveal a lot of information
> which users sometimes are not aware of; and I remember I informed you
> too when you posted a Base64 encrypted password of a live server.
>
> Synapse is not an academic project at all.. it never was. Multiple large
>
> health care providers in the US, the US Army, Governments in Asia and
> Europe, mobile resource management companies, one of the largest auto
> makers (who is still doing good in this economy), and SOA appliance
> vendor that embeds Synapse, European / Asian banks and a large media
> company across the US and Europe all use it. These are some of the users
>
> I know about.. and WSO2 would have more users too, that I am not aware
> of
>
> Here are some "public" comments by a few users in the past:
> "Thanks for the quick fix. Synapse continues to rock."
> <http://markmail.org/message/qbk4wvncjvtvat2s>
> "Synapse continues to impress. Looking forward to 1.1 features upon its
> release" <http://markmail.org/message/zwyrzleaop25bqad>
> "Thanks everyone for taking time to look into this -- you guys are great
>
> (know wonder Synapse is such a great product!)"
> <http://markmail.org/message/5lldrfesfb2wjv44>
> "I have been using the excellent Apache Synapse lightweight Enterprise
> Service Bus over the last year or so, including on a live, production
> project."
> <http://thejavamonkey.blogspot.com/2008/09/apache-synapse-how-to-run-wit
> hout.html>
>
> "We are using Apache Synapse version 1.1 to develop webservices for our
> mission critical "Surface Deployment and Distribution command" of U.S
> Army" <http://markmail.org/message/b67h6ggfh7vspt2u>
>
>>"I do not see the point of open source software if it cannot solve
> commercial problems;"
>
> I took this line out of the previous block, so that I can address it
> separately.. As far as I am aware, Open source software is not about
> building free software by volunteers to commercial companies who makes
> money out of it. For users its more about having lifetime access to the
> source code of the products, and knowing that licensing is free when you
>
> are ready to drop paying for support, so that you do not lock into
> proprietary vendor products, or with vendors who may not survive long
> term. When the code is available freely, you can compile and support it
> yourself, or get someone else to support it for you. In Apache, its
> about a community of people who are interested in building something..
> In Apache we even consider 'Community over code', and enjoy working with
>
> diverse people and learning from them for us to improve. But Apache does
>
> not pay us! we are free to sell support or consulting to make sure we
> also survive :-)
>
> Open source developers who volunteer their time, do not owe anything to
> the users - besides making sure the source code is available in full.
> Anyone who users the software is free to fix any issues, and if they
> like to contribute them back to the community.
>
>> Unfortunately not all Synapse users are expert enough to contribute
> yet; so
>> once again I mention the Catch 22 of �your comments. It takes time for
> new
>> people to get up to speed so they can contribute.
>>
>> So I spent about 4 hours researching this today before posting; and I
>> believe this is a contribution. We have real world requirements that
> would improve Synapse; this is a
>> contribution.
>>
>> Asking questions that do not seem to be covered by documentation; so
> that
>> others can benefit is a contribution. Thats all I can offer at the
> moment.
>>
> I am fully aware that all first time users would not be using Synapse
> like they were pro's. From my past experience at WSO2, and at present,
> many users ask for help on a POC and/or training to get them started.
> They then develop an initial prototype project themselves and/or move
> into production with development support or professional services.
> Thereafter they use dev support to ask questions and learn more advanced
>
> stuff.
>
> BTW, if I couldn't swim, I am aware there are many sites on the Internet
>
> that hosts loads of documentation, and even YouTube videos on how to
> swim all styles - free like open source software :-) . Now I may
> evaluate these techniques to see if they work in a pool of 5' depth..
> but not in the open sea, not without knowing someone is there to help me
>
> if I get in trouble. I would rather pay a professional trainer to train
> me to swim, and then try it out more stuff when I am comfortable.. and
> of course blog about it later so that others would learn something from
> me too. When I can swim and know I can stay afloat, the YouTube videos
> would allow me to learn more cool stuff.. and maybe I could invent
> something new and contribute it back so that others could follow it too.
>
> I will not be like Adinnapubbaka in the story Mattakundali [3] and try
> to learn and do what I cannot.
>> Unfortunately we have made an extensive investment in Synapse, in
> time, and
>> at this point I am trying not to have to drop the use of this product
> and
>> trying my hardest to demonstrate that this product can solve just one
> single
>> real world problem for us. Am I in a hurry, yes, we have passed the
> deadline.
>>
> So does this mean that you are trying to install Synapse into
> production?.. I would not invest much time or effort in an evaluation,
> but consider more options in parallel - maybe with some dev support to
> help me, to reduce my overall risk.
>
> cheers
> asankha
>
> [1] http://edocs.bea.com/alsb/docs261/sftptransport/intro.html
> [2] http://synapse.apache.org/Synapse_Samples.html#Sample254
> [3]
> http://books.google.lk/books?id=sTclW3OWyw8C&pg=PA37&dq=%22mattakundali+
> (Flat-Earring)%22
>
> --
> Asankha C. Perera
> AdroitLogic, http://adroitlogic.org
>
> http://esbmagic.blogspot.com
>
>
>
>
>
>

RE: Can VFS SFTP be used to download todays files only

[Kim Horn <ki...@icsglobal.net>]

I should note that we looked at using interval JMS queue and data base for resolving (5) below. An internal queue is a common soluion in Aqua Logic to do reliable messaging, when end point is down. The current solution uses local file system which is a less technically complex solution. The moves are not done but task completed status is stored in a DBMS, when transaction completes. We have to log this anyhow so was no real overhead.

Years ago we built an ESB for internal use in a large corporation, before ESBs were available as software. In that ESB we made reliable messaging a default function of the bus. It seems to be the base use case for any ESB, you always want a message to be guaranteed to be delivered. So no matter where in the ESB "script" the system failed it would start again in that same spot. If an end point was not available it would deliver later, given a certain number of re-tries.
We found the overhead of this, when it was made as a prime feature of the bus, was negligable.

Thanks
Kim


-----Original Message-----
From: Kim Horn [mailto:kim.horn@icsglobal.net]
Sent: Wed 08/04/2009 12:39
To: user@synapse.apache.org
Subject: RE: Can VFS SFTP be used to download todays files only
 
Hello Andreas,

Thanks for your concern. Yes I will do that, We have our own Jira for the project; and I will email the issues and a summary of the project as soon as we get over this hump. I am happy that we have not had to rule out Synapse totally and a hybrid approach may even be better; not sure ? 

I do have a list of Synapse Jira items, growing, that I will add, but I apologise for running out of time at the moment. When we get our current project up and running, I have allocated a week to work on Synapse stuff for you guys. 

In short and in relation to SFTP only the issues are:

1) we cannot have passwords in clear text in synapse scripts or anywhere. Due to the strict compliance we require; we can be seen as breaking US law doing this; one is the HIPPA Act. There are other compliance and accreditation standards we must comply too; that have very strict security policies. We review projects here so that we don't go to gaol (Jail in US) :-)
2) Ability to pass to VFS, parameters in variables (properties); so being able to download a file based on say a property value containing a name constructed, for example using a script or javainclude date.now().
3) I do not know if comments below are true but checking the 30 mb file limit; It seems this is a problem with Commons VFS SFTP and not Jsch. I am not sure what combination is used in Synapse VFS mediator. I know this has been addressed and I have tested as OK for VFS files.
4) upgrade to latest Jsch jar.
5) This one is a contentious issue: Roll back a VFS MOVE or DELETE, when the subsequent web service fails. This is like wrapping the SFTP get and WEB service put in a transaction. The issue we have now is that if a web service fails, the SFTP actions have been done as if successful. So the move has to be "compensated" by moving them back to the normal directory. Not sure if WS-Atomic transactions works in Synapse for SFTP; but that's the idea ? Not sure if transactions have been considered in general for Synapse ? I am not sure that this can be done using a failure sequence as just the move, without the download, on the SFTP server, is not possible ?

Of these only 1 to 3 impact us now, maybe 4.

Re 2: we are not happy that some partners do not allow us to move or delete files, but that's their policy and so what happens is that the files grow on their server and we currently use Date Modified to download latest. They also include the date of file in the file name. So we can use the Jsch lsat property getMTime() or build a file name regex using todays date to download new files. We currently have our system running in Australia and in the UK, we expanded to US and have found that SFTP is the most used B2B mechanism over there. We were Amazed. Basically in Australia Web Services are standard and no one uses SFTP, its dead. So this was the major attraction of Synapse to us.

It seems to me the base Use case for SFTP is to download a named file in text. Next is a regex. The next specialisation is to allow that name to be constructed programmatically. Next is to allow a regex to be constructed programmatically. More specialised use cases are the ones addressed by VFS mediator now where you use Move and Delete to get only new files only. For us it is rare to get this permission on other people servers. We do have one partner that is smart enough to do the move automatically, on their side, when we get a file. Other partners either allow us to delete or move files. So we have every combination.

Changing how VFS works now does not seem to be as easy, I guess as:
-  VFS Mediator may not be designed to be updated on the fly. I assume it would be only initialised on start up. The rqmt above requires the file regex to be changed each day, for the new day, or to be totally dynamic. Java regex, does not do date now; but it can be expanded to do this. I have some code for this.
-  It is not clear how you bootstrap this; as the property has to be set before a VFS proxy starts; hence why I avoided this solution now. This was my first experiment to try and reset the property for the regex after the first time the proxy executed Using:

<definitions xmlns="http://ws.apache.org/ns/synapse">
  <proxy name="StockQuoteProxy" transports="vfs">
	  <parameter name="transport.vfs.FileURI">file:///c:/test/in</parameter>
    <parameter name="transport.vfs.ContentType">text/plain</parameter>
    <parameter name="transport.vfs.FileNamePattern">.*\.edi</parameter>
    <parameter name="transport.PollInterval">15</parameter>
      <inSequence>
        <log level="full"/>
	<property name="transport.vfs.FileNamePattern" value=".*\.txt"/>
      </inSequence>
    </target>
  </proxy>
</definitions>

But this doesn't change the Pattern; only the ".edi" files get done :-)


Thanks all the best
Kim





-----Original Message-----
From: Andreas Veithen [mailto:andreas.veithen@gmail.com] 
Sent: Wednesday, 8 April 2009 7:13 AM
To: user@synapse.apache.org
Subject: Re: Can VFS SFTP be used to download todays files only

Kim,

Maybe at this point it would be useful if you could give a quick
overview of the open issues that you still face.

Andreas

On Tue, Apr 7, 2009 at 08:55, Kim Horn <ki...@icsglobal.net> wrote:
> Created a work around to this by writing an independent service using
> just Jsch & Quartz doing an sftp every hour.
> From Java we can pass in any expression we like and actually using Jsch
> can find the Latest Modified files from the system date. This also
> solves the SFTP passwords in clear text issue as now I can encrypt them
> in Java. So this service runs totally independent of Synapse. As a bonus
> this service can unzip SFTped zipped files we get from some locations
> and split large files as Jsch can tell us how big they are before
> SFTPing. A prior issue was submitting large files to some web services
> that required a new type of iterator in Synapse to be developed.
>
> So now Synapse just uses VFS to poles the file system, the service
> downloads files too, to submit manageable size text files to the
> destination Web Services.
>
> From the other Forums this solution also will resolve a possible issue
> with Commons VFS SFTP having problems with large files. Not sure if
> these issues are real but dates are fairly recent.
>
> Start Quotes:
>
> "This doesn't work with larger files. The VFS seems to have an issue
> with the getInputStream(), where it loads the whole file into memory
> before returning a memory stream. Eventually the JVM will run out of
> memory. In any case, it was a nice start--I'm going to try using Jsch to
> work around this problem.
> # posted by Blogger Wes : 11:34 PM, November 24, 2008
>
> Yes. The code is not working with larger files. It's OK up to 30mb
> files. If it exceeds more than 30mb it's causing problems(memory
> issues). Any idea how to get rid of this problem.
> # posted by Anonymous Anonymous : 3:17 PM, December 16, 2008.
>
> End Quotes.
>
>
> I have used a more recent release of Jsch than the Jar used in Synapse;
> that requires a later version of Java, so would not be possible to do
> this as a Synapse class mediator at the moment, although this would be a
> better solution.
>
> Thanks
> Kim
>
>
>
> -----Original Message-----
> From: Kim Horn [mailto:kim.horn@icsglobal.net]
> Sent: Tuesday, 7 April 2009 7:19 AM
> To: user@synapse.apache.org
> Subject: RE: Can VFS SFTP be used to download todays files only
>
> Hello Asankha,
>
> yes we did get some small simple scripts into production, to help our
> manual process, in parralel with evaluation, but had to pull them out as
> they did not run with the latest trunk required to solve another issue;
> the VFS memory not being released. The fix to this resulted in the "out"
> sequences producing exceptions with VFS.
>
> Thanks
> Kim
>
>
>
> -----Original Message-----
> From: Asankha Perera on behalf of Asankha C. Perera
> Sent: Mon 06/04/2009 20:20
> To: user@synapse.apache.org
> Subject: Re: Can VFS SFTP be used to download todays files only
>
> Hi Kim
>> 1) How do you specify a regex to download files with todays date that
> will
>> run in synapse unattended for weeks ?
>> Yes I can hard code an expression to match 31 March 2009 and after but
> that
>> is no good today, is it.
>> Java has the concept of Date NOW; this is what I mean by today.
>>
>> You say this is "Quite easy" please show me how, as I cannot see how
> this
>> can be done ? If it is easy then my problem is solved. Great, but this
> is my
>> question ?
>>
>> 2) �If an SFTP server has 2000 old files permanently on it now that
> will not
>> be removed; thats grows every day. How to I download just the new
> files ?
>> E.G Those placed on there with todays date in the name. I have no
> permission
>> on the partners SFTP server to MOVE files or DELETE them, they just
> stay
>> there. It is not a good assumption that files can be MOVED or DELETED
> on
>> another organisations SFTP server. Very rarely in real B2B do you get
> this
>> luxury.
>>
> The real world cases for which the transport has been originally written
>
> is to poll for a file or files at some specific location that matches
> some pattern, process them, and either 1) delete or 2) move them since
> the transport does not have state to remember what has been polled
> already.
>
> The commercial Aqualogic service bus follows this pattern too [1], while
>
> both Mule and Camel additionally supports the specification of date info
>
> in the file pattern. I couldn't readily find much about Fuse or JBoss
> etc.
>
> The solution I gave will not work, if you cannot move or delete
> processed files on the target sftp server, to prevent re-processing them
>
> later. However, supporting of date information in the pattern would be
> fairly easy to implement.
>> 3) You suggest SYnapse has a concept of new files ? How does this work
> ?
>>
> This is if you are creating a new file.. e.g. saving a message received
> via http/s, JMS, email etc as a file - maybe to an sftp location like in
> [2]
>> 4) Are you saying that when I start Synapse with an SFTP mediator it
>> remembers every file downloaded ever in the past; so that when we
> re-start
>> Synapse (after maintenence) it remembers the old files. If this is
> true this
>> is great. I am interested to know where this record of files is kept.
> Where
>> is the file ?
>>
>> 5) Otherwise I am missing some very interesting behaviour of the
> Synapse VFS
>> Mediator that does not seem to be provided by the samples or the
>> documentation or any prior Forum discussions.
>>
> No, the transport is implemented as whats called an "Axis2 transport" ..
>
> its actually out of the Synapse core, and not directly dependent or
> linked. Implementing a record of processed files - even if possible
> would not be worth it - I think the use of Date in the regex is the most
>
> simple way to handle this
>> I am quite happy to raise a Jira but there is no need if Synapse can
> do
>> this, easily, as you say ?
>>
> Well.. the point of a JIRA is that all good ideas are captured by those
> who propose them, so that new developers or the current developers -
> when they have free time, can look at real world user suggestions for
> improvement and implement them for future versions
>> I do not understand this comment about being in a hurry; we are
> evaluating
>> Synapse to solve a commercial problem and unfortunately we have
> encountered
>> many issues that we are trying to resolve.
> I interpreted your multiple emails in the past about dropping or almost
> dropping the use of Synapse to mean that you are in a hurry to get this
> implemented in Synapse, and thats why I even suggested a pure Java
> implementation. Of course if you are still in evaluation, the best is to
>
> try to select another few ESBs (open source and/or commercial) and try
> to implement the same, and consider the total cost and the complexity of
>
> the final solution. In this case, consider the effort to implement whats
>
> missing in Synapse into the total cost of the solution.
>
> However, from some of your emails, I see production requirements being
> talked about, and thats why sometime back I suggested getting commercial
>
> support - in which case, someone else would have taken care of all of
> these issues - implementing what was missing as required. To be honest,
> you could have got a complete solution for your real world problem, with
>
> commercial support and a guarantee, probably for much less than the
> number of hours you've already spent and would spend * a reasonable
> hourly rate for the cost of your time, which again is money.
>
>> Unfortunately when I think all the hurdles are solved; then we stumble
> over new ones.
>> Yes unfortunately real world projects have definite requirements and
> budgets and deadlines.
>> I do not see the point of open source software if it cannot solve
> commercial
>> problems; unless Synapse is just an Academic Research project ? I am
> sure
>> WS02 are using Synapse to solve real world commercial problems and not
> just
>> for research. I am sure they make money from using the code.
>>
> Of course, many real world users use Synapse - but some are shy to use
> the public forums, and risk exposing confidential information, security
> and passwords and sensitive data etc. I usually privately inform users
> whenever this happens.. since logs could reveal a lot of information
> which users sometimes are not aware of; and I remember I informed you
> too when you posted a Base64 encrypted password of a live server.
>
> Synapse is not an academic project at all.. it never was. Multiple large
>
> health care providers in the US, the US Army, Governments in Asia and
> Europe, mobile resource management companies, one of the largest auto
> makers (who is still doing good in this economy), and SOA appliance
> vendor that embeds Synapse, European / Asian banks and a large media
> company across the US and Europe all use it. These are some of the users
>
> I know about.. and WSO2 would have more users too, that I am not aware
> of
>
> Here are some "public" comments by a few users in the past:
> "Thanks for the quick fix. Synapse continues to rock."
> <http://markmail.org/message/qbk4wvncjvtvat2s>
> "Synapse continues to impress. Looking forward to 1.1 features upon its
> release" <http://markmail.org/message/zwyrzleaop25bqad>
> "Thanks everyone for taking time to look into this -- you guys are great
>
> (know wonder Synapse is such a great product!)"
> <http://markmail.org/message/5lldrfesfb2wjv44>
> "I have been using the excellent Apache Synapse lightweight Enterprise
> Service Bus over the last year or so, including on a live, production
> project."
> <http://thejavamonkey.blogspot.com/2008/09/apache-synapse-how-to-run-wit
> hout.html>
>
> "We are using Apache Synapse version 1.1 to develop webservices for our
> mission critical "Surface Deployment and Distribution command" of U.S
> Army" <http://markmail.org/message/b67h6ggfh7vspt2u>
>
>>"I do not see the point of open source software if it cannot solve
> commercial problems;"
>
> I took this line out of the previous block, so that I can address it
> separately.. As far as I am aware, Open source software is not about
> building free software by volunteers to commercial companies who makes
> money out of it. For users its more about having lifetime access to the
> source code of the products, and knowing that licensing is free when you
>
> are ready to drop paying for support, so that you do not lock into
> proprietary vendor products, or with vendors who may not survive long
> term. When the code is available freely, you can compile and support it
> yourself, or get someone else to support it for you. In Apache, its
> about a community of people who are interested in building something..
> In Apache we even consider 'Community over code', and enjoy working with
>
> diverse people and learning from them for us to improve. But Apache does
>
> not pay us! we are free to sell support or consulting to make sure we
> also survive :-)
>
> Open source developers who volunteer their time, do not owe anything to
> the users - besides making sure the source code is available in full.
> Anyone who users the software is free to fix any issues, and if they
> like to contribute them back to the community.
>
>> Unfortunately not all Synapse users are expert enough to contribute
> yet; so
>> once again I mention the Catch 22 of �your comments. It takes time for
> new
>> people to get up to speed so they can contribute.
>>
>> So I spent about 4 hours researching this today before posting; and I
>> believe this is a contribution. We have real world requirements that
> would improve Synapse; this is a
>> contribution.
>>
>> Asking questions that do not seem to be covered by documentation; so
> that
>> others can benefit is a contribution. Thats all I can offer at the
> moment.
>>
> I am fully aware that all first time users would not be using Synapse
> like they were pro's. From my past experience at WSO2, and at present,
> many users ask for help on a POC and/or training to get them started.
> They then develop an initial prototype project themselves and/or move
> into production with development support or professional services.
> Thereafter they use dev support to ask questions and learn more advanced
>
> stuff.
>
> BTW, if I couldn't swim, I am aware there are many sites on the Internet
>
> that hosts loads of documentation, and even YouTube videos on how to
> swim all styles - free like open source software :-) . Now I may
> evaluate these techniques to see if they work in a pool of 5' depth..
> but not in the open sea, not without knowing someone is there to help me
>
> if I get in trouble. I would rather pay a professional trainer to train
> me to swim, and then try it out more stuff when I am comfortable.. and
> of course blog about it later so that others would learn something from
> me too. When I can swim and know I can stay afloat, the YouTube videos
> would allow me to learn more cool stuff.. and maybe I could invent
> something new and contribute it back so that others could follow it too.
>
> I will not be like Adinnapubbaka in the story Mattakundali [3] and try
> to learn and do what I cannot.
>> Unfortunately we have made an extensive investment in Synapse, in
> time, and
>> at this point I am trying not to have to drop the use of this product
> and
>> trying my hardest to demonstrate that this product can solve just one
> single
>> real world problem for us. Am I in a hurry, yes, we have passed the
> deadline.
>>
> So does this mean that you are trying to install Synapse into
> production?.. I would not invest much time or effort in an evaluation,
> but consider more options in parallel - maybe with some dev support to
> help me, to reduce my overall risk.
>
> cheers
> asankha
>
> [1] http://edocs.bea.com/alsb/docs261/sftptransport/intro.html
> [2] http://synapse.apache.org/Synapse_Samples.html#Sample254
> [3]
> http://books.google.lk/books?id=sTclW3OWyw8C&pg=PA37&dq=%22mattakundali+
> (Flat-Earring)%22
>
> --
> Asankha C. Perera
> AdroitLogic, http://adroitlogic.org
>
> http://esbmagic.blogspot.com
>
>
>
>
>
>

RE: Can VFS SFTP be used to download todays files only

[Kim Horn <ki...@icsglobal.net>]

Hello Andreas,

Thanks for your concern. Yes I will do that, We have our own Jira for the project; and I will email the issues and a summary of the project as soon as we get over this hump. I am happy that we have not had to rule out Synapse totally and a hybrid approach may even be better; not sure ? 

I do have a list of Synapse Jira items, growing, that I will add, but I apologise for running out of time at the moment. When we get our current project up and running, I have allocated a week to work on Synapse stuff for you guys. 

In short and in relation to SFTP only the issues are:

1) we cannot have passwords in clear text in synapse scripts or anywhere. Due to the strict compliance we require; we can be seen as breaking US law doing this; one is the HIPPA Act. There are other compliance and accreditation standards we must comply too; that have very strict security policies. We review projects here so that we don't go to gaol (Jail in US) :-)
2) Ability to pass to VFS, parameters in variables (properties); so being able to download a file based on say a property value containing a name constructed, for example using a script or javainclude date.now().
3) I do not know if comments below are true but checking the 30 mb file limit; It seems this is a problem with Commons VFS SFTP and not Jsch. I am not sure what combination is used in Synapse VFS mediator. I know this has been addressed and I have tested as OK for VFS files.
4) upgrade to latest Jsch jar.
5) This one is a contentious issue: Roll back a VFS MOVE or DELETE, when the subsequent web service fails. This is like wrapping the SFTP get and WEB service put in a transaction. The issue we have now is that if a web service fails, the SFTP actions have been done as if successful. So the move has to be "compensated" by moving them back to the normal directory. Not sure if WS-Atomic transactions works in Synapse for SFTP; but that's the idea ? Not sure if transactions have been considered in general for Synapse ? I am not sure that this can be done using a failure sequence as just the move, without the download, on the SFTP server, is not possible ?

Of these only 1 to 3 impact us now, maybe 4.

Re 2: we are not happy that some partners do not allow us to move or delete files, but that's their policy and so what happens is that the files grow on their server and we currently use Date Modified to download latest. They also include the date of file in the file name. So we can use the Jsch lsat property getMTime() or build a file name regex using todays date to download new files. We currently have our system running in Australia and in the UK, we expanded to US and have found that SFTP is the most used B2B mechanism over there. We were Amazed. Basically in Australia Web Services are standard and no one uses SFTP, its dead. So this was the major attraction of Synapse to us.

It seems to me the base Use case for SFTP is to download a named file in text. Next is a regex. The next specialisation is to allow that name to be constructed programmatically. Next is to allow a regex to be constructed programmatically. More specialised use cases are the ones addressed by VFS mediator now where you use Move and Delete to get only new files only. For us it is rare to get this permission on other people servers. We do have one partner that is smart enough to do the move automatically, on their side, when we get a file. Other partners either allow us to delete or move files. So we have every combination.

Changing how VFS works now does not seem to be as easy, I guess as:
-  VFS Mediator may not be designed to be updated on the fly. I assume it would be only initialised on start up. The rqmt above requires the file regex to be changed each day, for the new day, or to be totally dynamic. Java regex, does not do date now; but it can be expanded to do this. I have some code for this.
-  It is not clear how you bootstrap this; as the property has to be set before a VFS proxy starts; hence why I avoided this solution now. This was my first experiment to try and reset the property for the regex after the first time the proxy executed Using:

<definitions xmlns="http://ws.apache.org/ns/synapse">
  <proxy name="StockQuoteProxy" transports="vfs">
	  <parameter name="transport.vfs.FileURI">file:///c:/test/in</parameter>
    <parameter name="transport.vfs.ContentType">text/plain</parameter>
    <parameter name="transport.vfs.FileNamePattern">.*\.edi</parameter>
    <parameter name="transport.PollInterval">15</parameter>
      <inSequence>
        <log level="full"/>
	<property name="transport.vfs.FileNamePattern" value=".*\.txt"/>
      </inSequence>
    </target>
  </proxy>
</definitions>

But this doesn't change the Pattern; only the ".edi" files get done :-)


Thanks all the best
Kim





-----Original Message-----
From: Andreas Veithen [mailto:andreas.veithen@gmail.com] 
Sent: Wednesday, 8 April 2009 7:13 AM
To: user@synapse.apache.org
Subject: Re: Can VFS SFTP be used to download todays files only

Kim,

Maybe at this point it would be useful if you could give a quick
overview of the open issues that you still face.

Andreas

On Tue, Apr 7, 2009 at 08:55, Kim Horn <ki...@icsglobal.net> wrote:
> Created a work around to this by writing an independent service using
> just Jsch & Quartz doing an sftp every hour.
> From Java we can pass in any expression we like and actually using Jsch
> can find the Latest Modified files from the system date. This also
> solves the SFTP passwords in clear text issue as now I can encrypt them
> in Java. So this service runs totally independent of Synapse. As a bonus
> this service can unzip SFTped zipped files we get from some locations
> and split large files as Jsch can tell us how big they are before
> SFTPing. A prior issue was submitting large files to some web services
> that required a new type of iterator in Synapse to be developed.
>
> So now Synapse just uses VFS to poles the file system, the service
> downloads files too, to submit manageable size text files to the
> destination Web Services.
>
> From the other Forums this solution also will resolve a possible issue
> with Commons VFS SFTP having problems with large files. Not sure if
> these issues are real but dates are fairly recent.
>
> Start Quotes:
>
> "This doesn't work with larger files. The VFS seems to have an issue
> with the getInputStream(), where it loads the whole file into memory
> before returning a memory stream. Eventually the JVM will run out of
> memory. In any case, it was a nice start--I'm going to try using Jsch to
> work around this problem.
> # posted by Blogger Wes : 11:34 PM, November 24, 2008
>
> Yes. The code is not working with larger files. It's OK up to 30mb
> files. If it exceeds more than 30mb it's causing problems(memory
> issues). Any idea how to get rid of this problem.
> # posted by Anonymous Anonymous : 3:17 PM, December 16, 2008.
>
> End Quotes.
>
>
> I have used a more recent release of Jsch than the Jar used in Synapse;
> that requires a later version of Java, so would not be possible to do
> this as a Synapse class mediator at the moment, although this would be a
> better solution.
>
> Thanks
> Kim
>
>
>
> -----Original Message-----
> From: Kim Horn [mailto:kim.horn@icsglobal.net]
> Sent: Tuesday, 7 April 2009 7:19 AM
> To: user@synapse.apache.org
> Subject: RE: Can VFS SFTP be used to download todays files only
>
> Hello Asankha,
>
> yes we did get some small simple scripts into production, to help our
> manual process, in parralel with evaluation, but had to pull them out as
> they did not run with the latest trunk required to solve another issue;
> the VFS memory not being released. The fix to this resulted in the "out"
> sequences producing exceptions with VFS.
>
> Thanks
> Kim
>
>
>
> -----Original Message-----
> From: Asankha Perera on behalf of Asankha C. Perera
> Sent: Mon 06/04/2009 20:20
> To: user@synapse.apache.org
> Subject: Re: Can VFS SFTP be used to download todays files only
>
> Hi Kim
>> 1) How do you specify a regex to download files with todays date that
> will
>> run in synapse unattended for weeks ?
>> Yes I can hard code an expression to match 31 March 2009 and after but
> that
>> is no good today, is it.
>> Java has the concept of Date NOW; this is what I mean by today.
>>
>> You say this is "Quite easy" please show me how, as I cannot see how
> this
>> can be done ? If it is easy then my problem is solved. Great, but this
> is my
>> question ?
>>
>> 2)  If an SFTP server has 2000 old files permanently on it now that
> will not
>> be removed; thats grows every day. How to I download just the new
> files ?
>> E.G Those placed on there with todays date in the name. I have no
> permission
>> on the partners SFTP server to MOVE files or DELETE them, they just
> stay
>> there. It is not a good assumption that files can be MOVED or DELETED
> on
>> another organisations SFTP server. Very rarely in real B2B do you get
> this
>> luxury.
>>
> The real world cases for which the transport has been originally written
>
> is to poll for a file or files at some specific location that matches
> some pattern, process them, and either 1) delete or 2) move them since
> the transport does not have state to remember what has been polled
> already.
>
> The commercial Aqualogic service bus follows this pattern too [1], while
>
> both Mule and Camel additionally supports the specification of date info
>
> in the file pattern. I couldn't readily find much about Fuse or JBoss
> etc.
>
> The solution I gave will not work, if you cannot move or delete
> processed files on the target sftp server, to prevent re-processing them
>
> later. However, supporting of date information in the pattern would be
> fairly easy to implement.
>> 3) You suggest SYnapse has a concept of new files ? How does this work
> ?
>>
> This is if you are creating a new file.. e.g. saving a message received
> via http/s, JMS, email etc as a file - maybe to an sftp location like in
> [2]
>> 4) Are you saying that when I start Synapse with an SFTP mediator it
>> remembers every file downloaded ever in the past; so that when we
> re-start
>> Synapse (after maintenence) it remembers the old files. If this is
> true this
>> is great. I am interested to know where this record of files is kept.
> Where
>> is the file ?
>>
>> 5) Otherwise I am missing some very interesting behaviour of the
> Synapse VFS
>> Mediator that does not seem to be provided by the samples or the
>> documentation or any prior Forum discussions.
>>
> No, the transport is implemented as whats called an "Axis2 transport" ..
>
> its actually out of the Synapse core, and not directly dependent or
> linked. Implementing a record of processed files - even if possible
> would not be worth it - I think the use of Date in the regex is the most
>
> simple way to handle this
>> I am quite happy to raise a Jira but there is no need if Synapse can
> do
>> this, easily, as you say ?
>>
> Well.. the point of a JIRA is that all good ideas are captured by those
> who propose them, so that new developers or the current developers -
> when they have free time, can look at real world user suggestions for
> improvement and implement them for future versions
>> I do not understand this comment about being in a hurry; we are
> evaluating
>> Synapse to solve a commercial problem and unfortunately we have
> encountered
>> many issues that we are trying to resolve.
> I interpreted your multiple emails in the past about dropping or almost
> dropping the use of Synapse to mean that you are in a hurry to get this
> implemented in Synapse, and thats why I even suggested a pure Java
> implementation. Of course if you are still in evaluation, the best is to
>
> try to select another few ESBs (open source and/or commercial) and try
> to implement the same, and consider the total cost and the complexity of
>
> the final solution. In this case, consider the effort to implement whats
>
> missing in Synapse into the total cost of the solution.
>
> However, from some of your emails, I see production requirements being
> talked about, and thats why sometime back I suggested getting commercial
>
> support - in which case, someone else would have taken care of all of
> these issues - implementing what was missing as required. To be honest,
> you could have got a complete solution for your real world problem, with
>
> commercial support and a guarantee, probably for much less than the
> number of hours you've already spent and would spend * a reasonable
> hourly rate for the cost of your time, which again is money.
>
>> Unfortunately when I think all the hurdles are solved; then we stumble
> over new ones.
>> Yes unfortunately real world projects have definite requirements and
> budgets and deadlines.
>> I do not see the point of open source software if it cannot solve
> commercial
>> problems; unless Synapse is just an Academic Research project ? I am
> sure
>> WS02 are using Synapse to solve real world commercial problems and not
> just
>> for research. I am sure they make money from using the code.
>>
> Of course, many real world users use Synapse - but some are shy to use
> the public forums, and risk exposing confidential information, security
> and passwords and sensitive data etc. I usually privately inform users
> whenever this happens.. since logs could reveal a lot of information
> which users sometimes are not aware of; and I remember I informed you
> too when you posted a Base64 encrypted password of a live server.
>
> Synapse is not an academic project at all.. it never was. Multiple large
>
> health care providers in the US, the US Army, Governments in Asia and
> Europe, mobile resource management companies, one of the largest auto
> makers (who is still doing good in this economy), and SOA appliance
> vendor that embeds Synapse, European / Asian banks and a large media
> company across the US and Europe all use it. These are some of the users
>
> I know about.. and WSO2 would have more users too, that I am not aware
> of
>
> Here are some "public" comments by a few users in the past:
> "Thanks for the quick fix. Synapse continues to rock."
> <http://markmail.org/message/qbk4wvncjvtvat2s>
> "Synapse continues to impress. Looking forward to 1.1 features upon its
> release" <http://markmail.org/message/zwyrzleaop25bqad>
> "Thanks everyone for taking time to look into this -- you guys are great
>
> (know wonder Synapse is such a great product!)"
> <http://markmail.org/message/5lldrfesfb2wjv44>
> "I have been using the excellent Apache Synapse lightweight Enterprise
> Service Bus over the last year or so, including on a live, production
> project."
> <http://thejavamonkey.blogspot.com/2008/09/apache-synapse-how-to-run-wit
> hout.html>
>
> "We are using Apache Synapse version 1.1 to develop webservices for our
> mission critical "Surface Deployment and Distribution command" of U.S
> Army" <http://markmail.org/message/b67h6ggfh7vspt2u>
>
>>"I do not see the point of open source software if it cannot solve
> commercial problems;"
>
> I took this line out of the previous block, so that I can address it
> separately.. As far as I am aware, Open source software is not about
> building free software by volunteers to commercial companies who makes
> money out of it. For users its more about having lifetime access to the
> source code of the products, and knowing that licensing is free when you
>
> are ready to drop paying for support, so that you do not lock into
> proprietary vendor products, or with vendors who may not survive long
> term. When the code is available freely, you can compile and support it
> yourself, or get someone else to support it for you. In Apache, its
> about a community of people who are interested in building something..
> In Apache we even consider 'Community over code', and enjoy working with
>
> diverse people and learning from them for us to improve. But Apache does
>
> not pay us! we are free to sell support or consulting to make sure we
> also survive :-)
>
> Open source developers who volunteer their time, do not owe anything to
> the users - besides making sure the source code is available in full.
> Anyone who users the software is free to fix any issues, and if they
> like to contribute them back to the community.
>
>> Unfortunately not all Synapse users are expert enough to contribute
> yet; so
>> once again I mention the Catch 22 of  your comments. It takes time for
> new
>> people to get up to speed so they can contribute.
>>
>> So I spent about 4 hours researching this today before posting; and I
>> believe this is a contribution. We have real world requirements that
> would improve Synapse; this is a
>> contribution.
>>
>> Asking questions that do not seem to be covered by documentation; so
> that
>> others can benefit is a contribution. Thats all I can offer at the
> moment.
>>
> I am fully aware that all first time users would not be using Synapse
> like they were pro's. From my past experience at WSO2, and at present,
> many users ask for help on a POC and/or training to get them started.
> They then develop an initial prototype project themselves and/or move
> into production with development support or professional services.
> Thereafter they use dev support to ask questions and learn more advanced
>
> stuff.
>
> BTW, if I couldn't swim, I am aware there are many sites on the Internet
>
> that hosts loads of documentation, and even YouTube videos on how to
> swim all styles - free like open source software :-) . Now I may
> evaluate these techniques to see if they work in a pool of 5' depth..
> but not in the open sea, not without knowing someone is there to help me
>
> if I get in trouble. I would rather pay a professional trainer to train
> me to swim, and then try it out more stuff when I am comfortable.. and
> of course blog about it later so that others would learn something from
> me too. When I can swim and know I can stay afloat, the YouTube videos
> would allow me to learn more cool stuff.. and maybe I could invent
> something new and contribute it back so that others could follow it too.
>
> I will not be like Adinnapubbaka in the story Mattakundali [3] and try
> to learn and do what I cannot.
>> Unfortunately we have made an extensive investment in Synapse, in
> time, and
>> at this point I am trying not to have to drop the use of this product
> and
>> trying my hardest to demonstrate that this product can solve just one
> single
>> real world problem for us. Am I in a hurry, yes, we have passed the
> deadline.
>>
> So does this mean that you are trying to install Synapse into
> production?.. I would not invest much time or effort in an evaluation,
> but consider more options in parallel - maybe with some dev support to
> help me, to reduce my overall risk.
>
> cheers
> asankha
>
> [1] http://edocs.bea.com/alsb/docs261/sftptransport/intro.html
> [2] http://synapse.apache.org/Synapse_Samples.html#Sample254
> [3]
> http://books.google.lk/books?id=sTclW3OWyw8C&pg=PA37&dq=%22mattakundali+
> (Flat-Earring)%22
>
> --
> Asankha C. Perera
> AdroitLogic, http://adroitlogic.org
>
> http://esbmagic.blogspot.com
>
>
>
>
>
>

Re: Can VFS SFTP be used to download todays files only

[Andreas Veithen <an...@gmail.com>]

Kim,

Maybe at this point it would be useful if you could give a quick
overview of the open issues that you still face.

Andreas

On Tue, Apr 7, 2009 at 08:55, Kim Horn <ki...@icsglobal.net> wrote:
> Created a work around to this by writing an independent service using
> just Jsch & Quartz doing an sftp every hour.
> From Java we can pass in any expression we like and actually using Jsch
> can find the Latest Modified files from the system date. This also
> solves the SFTP passwords in clear text issue as now I can encrypt them
> in Java. So this service runs totally independent of Synapse. As a bonus
> this service can unzip SFTped zipped files we get from some locations
> and split large files as Jsch can tell us how big they are before
> SFTPing. A prior issue was submitting large files to some web services
> that required a new type of iterator in Synapse to be developed.
>
> So now Synapse just uses VFS to poles the file system, the service
> downloads files too, to submit manageable size text files to the
> destination Web Services.
>
> From the other Forums this solution also will resolve a possible issue
> with Commons VFS SFTP having problems with large files. Not sure if
> these issues are real but dates are fairly recent.
>
> Start Quotes:
>
> "This doesn't work with larger files. The VFS seems to have an issue
> with the getInputStream(), where it loads the whole file into memory
> before returning a memory stream. Eventually the JVM will run out of
> memory. In any case, it was a nice start--I'm going to try using Jsch to
> work around this problem.
> # posted by Blogger Wes : 11:34 PM, November 24, 2008
>
> Yes. The code is not working with larger files. It's OK up to 30mb
> files. If it exceeds more than 30mb it's causing problems(memory
> issues). Any idea how to get rid of this problem.
> # posted by Anonymous Anonymous : 3:17 PM, December 16, 2008.
>
> End Quotes.
>
>
> I have used a more recent release of Jsch than the Jar used in Synapse;
> that requires a later version of Java, so would not be possible to do
> this as a Synapse class mediator at the moment, although this would be a
> better solution.
>
> Thanks
> Kim
>
>
>
> -----Original Message-----
> From: Kim Horn [mailto:kim.horn@icsglobal.net]
> Sent: Tuesday, 7 April 2009 7:19 AM
> To: user@synapse.apache.org
> Subject: RE: Can VFS SFTP be used to download todays files only
>
> Hello Asankha,
>
> yes we did get some small simple scripts into production, to help our
> manual process, in parralel with evaluation, but had to pull them out as
> they did not run with the latest trunk required to solve another issue;
> the VFS memory not being released. The fix to this resulted in the "out"
> sequences producing exceptions with VFS.
>
> Thanks
> Kim
>
>
>
> -----Original Message-----
> From: Asankha Perera on behalf of Asankha C. Perera
> Sent: Mon 06/04/2009 20:20
> To: user@synapse.apache.org
> Subject: Re: Can VFS SFTP be used to download todays files only
>
> Hi Kim
>> 1) How do you specify a regex to download files with todays date that
> will
>> run in synapse unattended for weeks ?
>> Yes I can hard code an expression to match 31 March 2009 and after but
> that
>> is no good today, is it.
>> Java has the concept of Date NOW; this is what I mean by today.
>>
>> You say this is "Quite easy" please show me how, as I cannot see how
> this
>> can be done ? If it is easy then my problem is solved. Great, but this
> is my
>> question ?
>>
>> 2)  If an SFTP server has 2000 old files permanently on it now that
> will not
>> be removed; thats grows every day. How to I download just the new
> files ?
>> E.G Those placed on there with todays date in the name. I have no
> permission
>> on the partners SFTP server to MOVE files or DELETE them, they just
> stay
>> there. It is not a good assumption that files can be MOVED or DELETED
> on
>> another organisations SFTP server. Very rarely in real B2B do you get
> this
>> luxury.
>>
> The real world cases for which the transport has been originally written
>
> is to poll for a file or files at some specific location that matches
> some pattern, process them, and either 1) delete or 2) move them since
> the transport does not have state to remember what has been polled
> already.
>
> The commercial Aqualogic service bus follows this pattern too [1], while
>
> both Mule and Camel additionally supports the specification of date info
>
> in the file pattern. I couldn't readily find much about Fuse or JBoss
> etc.
>
> The solution I gave will not work, if you cannot move or delete
> processed files on the target sftp server, to prevent re-processing them
>
> later. However, supporting of date information in the pattern would be
> fairly easy to implement.
>> 3) You suggest SYnapse has a concept of new files ? How does this work
> ?
>>
> This is if you are creating a new file.. e.g. saving a message received
> via http/s, JMS, email etc as a file - maybe to an sftp location like in
> [2]
>> 4) Are you saying that when I start Synapse with an SFTP mediator it
>> remembers every file downloaded ever in the past; so that when we
> re-start
>> Synapse (after maintenence) it remembers the old files. If this is
> true this
>> is great. I am interested to know where this record of files is kept.
> Where
>> is the file ?
>>
>> 5) Otherwise I am missing some very interesting behaviour of the
> Synapse VFS
>> Mediator that does not seem to be provided by the samples or the
>> documentation or any prior Forum discussions.
>>
> No, the transport is implemented as whats called an "Axis2 transport" ..
>
> its actually out of the Synapse core, and not directly dependent or
> linked. Implementing a record of processed files - even if possible
> would not be worth it - I think the use of Date in the regex is the most
>
> simple way to handle this
>> I am quite happy to raise a Jira but there is no need if Synapse can
> do
>> this, easily, as you say ?
>>
> Well.. the point of a JIRA is that all good ideas are captured by those
> who propose them, so that new developers or the current developers -
> when they have free time, can look at real world user suggestions for
> improvement and implement them for future versions
>> I do not understand this comment about being in a hurry; we are
> evaluating
>> Synapse to solve a commercial problem and unfortunately we have
> encountered
>> many issues that we are trying to resolve.
> I interpreted your multiple emails in the past about dropping or almost
> dropping the use of Synapse to mean that you are in a hurry to get this
> implemented in Synapse, and thats why I even suggested a pure Java
> implementation. Of course if you are still in evaluation, the best is to
>
> try to select another few ESBs (open source and/or commercial) and try
> to implement the same, and consider the total cost and the complexity of
>
> the final solution. In this case, consider the effort to implement whats
>
> missing in Synapse into the total cost of the solution.
>
> However, from some of your emails, I see production requirements being
> talked about, and thats why sometime back I suggested getting commercial
>
> support - in which case, someone else would have taken care of all of
> these issues - implementing what was missing as required. To be honest,
> you could have got a complete solution for your real world problem, with
>
> commercial support and a guarantee, probably for much less than the
> number of hours you've already spent and would spend * a reasonable
> hourly rate for the cost of your time, which again is money.
>
>> Unfortunately when I think all the hurdles are solved; then we stumble
> over new ones.
>> Yes unfortunately real world projects have definite requirements and
> budgets and deadlines.
>> I do not see the point of open source software if it cannot solve
> commercial
>> problems; unless Synapse is just an Academic Research project ? I am
> sure
>> WS02 are using Synapse to solve real world commercial problems and not
> just
>> for research. I am sure they make money from using the code.
>>
> Of course, many real world users use Synapse - but some are shy to use
> the public forums, and risk exposing confidential information, security
> and passwords and sensitive data etc. I usually privately inform users
> whenever this happens.. since logs could reveal a lot of information
> which users sometimes are not aware of; and I remember I informed you
> too when you posted a Base64 encrypted password of a live server.
>
> Synapse is not an academic project at all.. it never was. Multiple large
>
> health care providers in the US, the US Army, Governments in Asia and
> Europe, mobile resource management companies, one of the largest auto
> makers (who is still doing good in this economy), and SOA appliance
> vendor that embeds Synapse, European / Asian banks and a large media
> company across the US and Europe all use it. These are some of the users
>
> I know about.. and WSO2 would have more users too, that I am not aware
> of
>
> Here are some "public" comments by a few users in the past:
> "Thanks for the quick fix. Synapse continues to rock."
> <http://markmail.org/message/qbk4wvncjvtvat2s>
> "Synapse continues to impress. Looking forward to 1.1 features upon its
> release" <http://markmail.org/message/zwyrzleaop25bqad>
> "Thanks everyone for taking time to look into this -- you guys are great
>
> (know wonder Synapse is such a great product!)"
> <http://markmail.org/message/5lldrfesfb2wjv44>
> "I have been using the excellent Apache Synapse lightweight Enterprise
> Service Bus over the last year or so, including on a live, production
> project."
> <http://thejavamonkey.blogspot.com/2008/09/apache-synapse-how-to-run-wit
> hout.html>
>
> "We are using Apache Synapse version 1.1 to develop webservices for our
> mission critical "Surface Deployment and Distribution command" of U.S
> Army" <http://markmail.org/message/b67h6ggfh7vspt2u>
>
>>"I do not see the point of open source software if it cannot solve
> commercial problems;"
>
> I took this line out of the previous block, so that I can address it
> separately.. As far as I am aware, Open source software is not about
> building free software by volunteers to commercial companies who makes
> money out of it. For users its more about having lifetime access to the
> source code of the products, and knowing that licensing is free when you
>
> are ready to drop paying for support, so that you do not lock into
> proprietary vendor products, or with vendors who may not survive long
> term. When the code is available freely, you can compile and support it
> yourself, or get someone else to support it for you. In Apache, its
> about a community of people who are interested in building something..
> In Apache we even consider 'Community over code', and enjoy working with
>
> diverse people and learning from them for us to improve. But Apache does
>
> not pay us! we are free to sell support or consulting to make sure we
> also survive :-)
>
> Open source developers who volunteer their time, do not owe anything to
> the users - besides making sure the source code is available in full.
> Anyone who users the software is free to fix any issues, and if they
> like to contribute them back to the community.
>
>> Unfortunately not all Synapse users are expert enough to contribute
> yet; so
>> once again I mention the Catch 22 of  your comments. It takes time for
> new
>> people to get up to speed so they can contribute.
>>
>> So I spent about 4 hours researching this today before posting; and I
>> believe this is a contribution. We have real world requirements that
> would improve Synapse; this is a
>> contribution.
>>
>> Asking questions that do not seem to be covered by documentation; so
> that
>> others can benefit is a contribution. Thats all I can offer at the
> moment.
>>
> I am fully aware that all first time users would not be using Synapse
> like they were pro's. From my past experience at WSO2, and at present,
> many users ask for help on a POC and/or training to get them started.
> They then develop an initial prototype project themselves and/or move
> into production with development support or professional services.
> Thereafter they use dev support to ask questions and learn more advanced
>
> stuff.
>
> BTW, if I couldn't swim, I am aware there are many sites on the Internet
>
> that hosts loads of documentation, and even YouTube videos on how to
> swim all styles - free like open source software :-) . Now I may
> evaluate these techniques to see if they work in a pool of 5' depth..
> but not in the open sea, not without knowing someone is there to help me
>
> if I get in trouble. I would rather pay a professional trainer to train
> me to swim, and then try it out more stuff when I am comfortable.. and
> of course blog about it later so that others would learn something from
> me too. When I can swim and know I can stay afloat, the YouTube videos
> would allow me to learn more cool stuff.. and maybe I could invent
> something new and contribute it back so that others could follow it too.
>
> I will not be like Adinnapubbaka in the story Mattakundali [3] and try
> to learn and do what I cannot.
>> Unfortunately we have made an extensive investment in Synapse, in
> time, and
>> at this point I am trying not to have to drop the use of this product
> and
>> trying my hardest to demonstrate that this product can solve just one
> single
>> real world problem for us. Am I in a hurry, yes, we have passed the
> deadline.
>>
> So does this mean that you are trying to install Synapse into
> production?.. I would not invest much time or effort in an evaluation,
> but consider more options in parallel - maybe with some dev support to
> help me, to reduce my overall risk.
>
> cheers
> asankha
>
> [1] http://edocs.bea.com/alsb/docs261/sftptransport/intro.html
> [2] http://synapse.apache.org/Synapse_Samples.html#Sample254
> [3]
> http://books.google.lk/books?id=sTclW3OWyw8C&pg=PA37&dq=%22mattakundali+
> (Flat-Earring)%22
>
> --
> Asankha C. Perera
> AdroitLogic, http://adroitlogic.org
>
> http://esbmagic.blogspot.com
>
>
>
>
>
>

RE: Can VFS SFTP be used to download todays files only

[Kim Horn <ki...@icsglobal.net>]

Created a work around to this by writing an independent service using
just Jsch & Quartz doing an sftp every hour.
>From Java we can pass in any expression we like and actually using Jsch
can find the Latest Modified files from the system date. This also
solves the SFTP passwords in clear text issue as now I can encrypt them
in Java. So this service runs totally independent of Synapse. As a bonus
this service can unzip SFTped zipped files we get from some locations
and split large files as Jsch can tell us how big they are before
SFTPing. A prior issue was submitting large files to some web services
that required a new type of iterator in Synapse to be developed.

So now Synapse just uses VFS to poles the file system, the service
downloads files too, to submit manageable size text files to the
destination Web Services.

>From the other Forums this solution also will resolve a possible issue
with Commons VFS SFTP having problems with large files. Not sure if
these issues are real but dates are fairly recent.

Start Quotes:

"This doesn't work with larger files. The VFS seems to have an issue
with the getInputStream(), where it loads the whole file into memory
before returning a memory stream. Eventually the JVM will run out of
memory. In any case, it was a nice start--I'm going to try using Jsch to
work around this problem.
# posted by Blogger Wes : 11:34 PM, November 24, 2008
 
Yes. The code is not working with larger files. It's OK up to 30mb
files. If it exceeds more than 30mb it's causing problems(memory
issues). Any idea how to get rid of this problem.
# posted by Anonymous Anonymous : 3:17 PM, December 16, 2008.

End Quotes.


I have used a more recent release of Jsch than the Jar used in Synapse;
that requires a later version of Java, so would not be possible to do
this as a Synapse class mediator at the moment, although this would be a
better solution. 

Thanks
Kim



-----Original Message-----
From: Kim Horn [mailto:kim.horn@icsglobal.net] 
Sent: Tuesday, 7 April 2009 7:19 AM
To: user@synapse.apache.org
Subject: RE: Can VFS SFTP be used to download todays files only

Hello Asankha,

yes we did get some small simple scripts into production, to help our
manual process, in parralel with evaluation, but had to pull them out as
they did not run with the latest trunk required to solve another issue;
the VFS memory not being released. The fix to this resulted in the "out"
sequences producing exceptions with VFS.

Thanks
Kim



-----Original Message-----
From: Asankha Perera on behalf of Asankha C. Perera
Sent: Mon 06/04/2009 20:20
To: user@synapse.apache.org
Subject: Re: Can VFS SFTP be used to download todays files only
 
Hi Kim
> 1) How do you specify a regex to download files with todays date that
will
> run in synapse unattended for weeks ?
> Yes I can hard code an expression to match 31 March 2009 and after but
that
> is no good today, is it. 
> Java has the concept of Date NOW; this is what I mean by today. 
>
> You say this is "Quite easy" please show me how, as I cannot see how
this
> can be done ? If it is easy then my problem is solved. Great, but this
is my
> question ?
>
> 2)  If an SFTP server has 2000 old files permanently on it now that
will not
> be removed; thats grows every day. How to I download just the new
files ?
> E.G Those placed on there with todays date in the name. I have no
permission
> on the partners SFTP server to MOVE files or DELETE them, they just
stay
> there. It is not a good assumption that files can be MOVED or DELETED
on
> another organisations SFTP server. Very rarely in real B2B do you get
this
> luxury.
>   
The real world cases for which the transport has been originally written

is to poll for a file or files at some specific location that matches 
some pattern, process them, and either 1) delete or 2) move them since 
the transport does not have state to remember what has been polled
already.

The commercial Aqualogic service bus follows this pattern too [1], while

both Mule and Camel additionally supports the specification of date info

in the file pattern. I couldn't readily find much about Fuse or JBoss
etc.

The solution I gave will not work, if you cannot move or delete 
processed files on the target sftp server, to prevent re-processing them

later. However, supporting of date information in the pattern would be 
fairly easy to implement.
> 3) You suggest SYnapse has a concept of new files ? How does this work
?
>   
This is if you are creating a new file.. e.g. saving a message received 
via http/s, JMS, email etc as a file - maybe to an sftp location like in
[2]
> 4) Are you saying that when I start Synapse with an SFTP mediator it
> remembers every file downloaded ever in the past; so that when we
re-start
> Synapse (after maintenence) it remembers the old files. If this is
true this
> is great. I am interested to know where this record of files is kept.
Where
> is the file ?
>
> 5) Otherwise I am missing some very interesting behaviour of the
Synapse VFS
> Mediator that does not seem to be provided by the samples or the
> documentation or any prior Forum discussions.
>   
No, the transport is implemented as whats called an "Axis2 transport" ..

its actually out of the Synapse core, and not directly dependent or 
linked. Implementing a record of processed files - even if possible 
would not be worth it - I think the use of Date in the regex is the most

simple way to handle this
> I am quite happy to raise a Jira but there is no need if Synapse can
do
> this, easily, as you say ? 
>   
Well.. the point of a JIRA is that all good ideas are captured by those 
who propose them, so that new developers or the current developers - 
when they have free time, can look at real world user suggestions for 
improvement and implement them for future versions
> I do not understand this comment about being in a hurry; we are
evaluating
> Synapse to solve a commercial problem and unfortunately we have
encountered
> many issues that we are trying to resolve.
I interpreted your multiple emails in the past about dropping or almost 
dropping the use of Synapse to mean that you are in a hurry to get this 
implemented in Synapse, and thats why I even suggested a pure Java 
implementation. Of course if you are still in evaluation, the best is to

try to select another few ESBs (open source and/or commercial) and try 
to implement the same, and consider the total cost and the complexity of

the final solution. In this case, consider the effort to implement whats

missing in Synapse into the total cost of the solution.

However, from some of your emails, I see production requirements being 
talked about, and thats why sometime back I suggested getting commercial

support - in which case, someone else would have taken care of all of 
these issues - implementing what was missing as required. To be honest, 
you could have got a complete solution for your real world problem, with

commercial support and a guarantee, probably for much less than the 
number of hours you've already spent and would spend * a reasonable 
hourly rate for the cost of your time, which again is money.

> Unfortunately when I think all the hurdles are solved; then we stumble
over new ones.
> Yes unfortunately real world projects have definite requirements and
budgets and deadlines.
> I do not see the point of open source software if it cannot solve
commercial
> problems; unless Synapse is just an Academic Research project ? I am
sure
> WS02 are using Synapse to solve real world commercial problems and not
just
> for research. I am sure they make money from using the code.
>   
Of course, many real world users use Synapse - but some are shy to use 
the public forums, and risk exposing confidential information, security 
and passwords and sensitive data etc. I usually privately inform users 
whenever this happens.. since logs could reveal a lot of information 
which users sometimes are not aware of; and I remember I informed you 
too when you posted a Base64 encrypted password of a live server.

Synapse is not an academic project at all.. it never was. Multiple large

health care providers in the US, the US Army, Governments in Asia and 
Europe, mobile resource management companies, one of the largest auto 
makers (who is still doing good in this economy), and SOA appliance 
vendor that embeds Synapse, European / Asian banks and a large media 
company across the US and Europe all use it. These are some of the users

I know about.. and WSO2 would have more users too, that I am not aware
of

Here are some "public" comments by a few users in the past:
"Thanks for the quick fix. Synapse continues to rock." 
<http://markmail.org/message/qbk4wvncjvtvat2s>
"Synapse continues to impress. Looking forward to 1.1 features upon its 
release" <http://markmail.org/message/zwyrzleaop25bqad>
"Thanks everyone for taking time to look into this -- you guys are great

(know wonder Synapse is such a great product!)" 
<http://markmail.org/message/5lldrfesfb2wjv44>
"I have been using the excellent Apache Synapse lightweight Enterprise 
Service Bus over the last year or so, including on a live, production 
project." 
<http://thejavamonkey.blogspot.com/2008/09/apache-synapse-how-to-run-wit
hout.html> 

"We are using Apache Synapse version 1.1 to develop webservices for our 
mission critical "Surface Deployment and Distribution command" of U.S 
Army" <http://markmail.org/message/b67h6ggfh7vspt2u>

>"I do not see the point of open source software if it cannot solve
commercial problems;"

I took this line out of the previous block, so that I can address it 
separately.. As far as I am aware, Open source software is not about 
building free software by volunteers to commercial companies who makes 
money out of it. For users its more about having lifetime access to the 
source code of the products, and knowing that licensing is free when you

are ready to drop paying for support, so that you do not lock into 
proprietary vendor products, or with vendors who may not survive long 
term. When the code is available freely, you can compile and support it 
yourself, or get someone else to support it for you. In Apache, its 
about a community of people who are interested in building something.. 
In Apache we even consider 'Community over code', and enjoy working with

diverse people and learning from them for us to improve. But Apache does

not pay us! we are free to sell support or consulting to make sure we 
also survive :-)

Open source developers who volunteer their time, do not owe anything to 
the users - besides making sure the source code is available in full. 
Anyone who users the software is free to fix any issues, and if they 
like to contribute them back to the community.

> Unfortunately not all Synapse users are expert enough to contribute
yet; so
> once again I mention the Catch 22 of  your comments. It takes time for
new
> people to get up to speed so they can contribute.
>
> So I spent about 4 hours researching this today before posting; and I
> believe this is a contribution. We have real world requirements that
would improve Synapse; this is a
> contribution.
>
> Asking questions that do not seem to be covered by documentation; so
that
> others can benefit is a contribution. Thats all I can offer at the
moment.
>   
I am fully aware that all first time users would not be using Synapse 
like they were pro's. From my past experience at WSO2, and at present, 
many users ask for help on a POC and/or training to get them started. 
They then develop an initial prototype project themselves and/or move 
into production with development support or professional services. 
Thereafter they use dev support to ask questions and learn more advanced

stuff.

BTW, if I couldn't swim, I am aware there are many sites on the Internet

that hosts loads of documentation, and even YouTube videos on how to 
swim all styles - free like open source software :-) . Now I may 
evaluate these techniques to see if they work in a pool of 5' depth.. 
but not in the open sea, not without knowing someone is there to help me

if I get in trouble. I would rather pay a professional trainer to train 
me to swim, and then try it out more stuff when I am comfortable.. and 
of course blog about it later so that others would learn something from 
me too. When I can swim and know I can stay afloat, the YouTube videos 
would allow me to learn more cool stuff.. and maybe I could invent 
something new and contribute it back so that others could follow it too.

I will not be like Adinnapubbaka in the story Mattakundali [3] and try 
to learn and do what I cannot.
> Unfortunately we have made an extensive investment in Synapse, in
time, and
> at this point I am trying not to have to drop the use of this product
and
> trying my hardest to demonstrate that this product can solve just one
single
> real world problem for us. Am I in a hurry, yes, we have passed the
deadline.
>   
So does this mean that you are trying to install Synapse into 
production?.. I would not invest much time or effort in an evaluation, 
but consider more options in parallel - maybe with some dev support to 
help me, to reduce my overall risk.

cheers
asankha

[1] http://edocs.bea.com/alsb/docs261/sftptransport/intro.html
[2] http://synapse.apache.org/Synapse_Samples.html#Sample254
[3] 
http://books.google.lk/books?id=sTclW3OWyw8C&pg=PA37&dq=%22mattakundali+
(Flat-Earring)%22

-- 
Asankha C. Perera
AdroitLogic, http://adroitlogic.org

http://esbmagic.blogspot.com

RE: Can VFS SFTP be used to download todays files only

[Kim Horn <ki...@icsglobal.net>]

Hello Asankha,

yes we did get some small simple scripts into production, to help our manual process, in parralel with evaluation, but had to pull them out as they did not run with the latest trunk required to solve another issue; the VFS memory not being released. The fix to this resulted in the "out" sequences producing exceptions with VFS.

Thanks
Kim



-----Original Message-----
From: Asankha Perera on behalf of Asankha C. Perera
Sent: Mon 06/04/2009 20:20
To: user@synapse.apache.org
Subject: Re: Can VFS SFTP be used to download todays files only
 
Hi Kim
> 1) How do you specify a regex to download files with todays date that will
> run in synapse unattended for weeks ?
> Yes I can hard code an expression to match 31 March 2009 and after but that
> is no good today, is it. 
> Java has the concept of Date NOW; this is what I mean by today. 
>
> You say this is "Quite easy" please show me how, as I cannot see how this
> can be done ? If it is easy then my problem is solved. Great, but this is my
> question ?
>
> 2)  If an SFTP server has 2000 old files permanently on it now that will not
> be removed; thats grows every day. How to I download just the new files ?
> E.G Those placed on there with todays date in the name. I have no permission
> on the partners SFTP server to MOVE files or DELETE them, they just stay
> there. It is not a good assumption that files can be MOVED or DELETED on
> another organisations SFTP server. Very rarely in real B2B do you get this
> luxury.
>   
The real world cases for which the transport has been originally written 
is to poll for a file or files at some specific location that matches 
some pattern, process them, and either 1) delete or 2) move them since 
the transport does not have state to remember what has been polled already.

The commercial Aqualogic service bus follows this pattern too [1], while 
both Mule and Camel additionally supports the specification of date info 
in the file pattern. I couldn't readily find much about Fuse or JBoss etc.

The solution I gave will not work, if you cannot move or delete 
processed files on the target sftp server, to prevent re-processing them 
later. However, supporting of date information in the pattern would be 
fairly easy to implement.
> 3) You suggest SYnapse has a concept of new files ? How does this work ?
>   
This is if you are creating a new file.. e.g. saving a message received 
via http/s, JMS, email etc as a file - maybe to an sftp location like in [2]
> 4) Are you saying that when I start Synapse with an SFTP mediator it
> remembers every file downloaded ever in the past; so that when we re-start
> Synapse (after maintenence) it remembers the old files. If this is true this
> is great. I am interested to know where this record of files is kept. Where
> is the file ?
>
> 5) Otherwise I am missing some very interesting behaviour of the Synapse VFS
> Mediator that does not seem to be provided by the samples or the
> documentation or any prior Forum discussions.
>   
No, the transport is implemented as whats called an "Axis2 transport" .. 
its actually out of the Synapse core, and not directly dependent or 
linked. Implementing a record of processed files - even if possible 
would not be worth it - I think the use of Date in the regex is the most 
simple way to handle this
> I am quite happy to raise a Jira but there is no need if Synapse can do
> this, easily, as you say ? 
>   
Well.. the point of a JIRA is that all good ideas are captured by those 
who propose them, so that new developers or the current developers - 
when they have free time, can look at real world user suggestions for 
improvement and implement them for future versions
> I do not understand this comment about being in a hurry; we are evaluating
> Synapse to solve a commercial problem and unfortunately we have encountered
> many issues that we are trying to resolve.
I interpreted your multiple emails in the past about dropping or almost 
dropping the use of Synapse to mean that you are in a hurry to get this 
implemented in Synapse, and thats why I even suggested a pure Java 
implementation. Of course if you are still in evaluation, the best is to 
try to select another few ESBs (open source and/or commercial) and try 
to implement the same, and consider the total cost and the complexity of 
the final solution. In this case, consider the effort to implement whats 
missing in Synapse into the total cost of the solution.

However, from some of your emails, I see production requirements being 
talked about, and thats why sometime back I suggested getting commercial 
support - in which case, someone else would have taken care of all of 
these issues - implementing what was missing as required. To be honest, 
you could have got a complete solution for your real world problem, with 
commercial support and a guarantee, probably for much less than the 
number of hours you've already spent and would spend * a reasonable 
hourly rate for the cost of your time, which again is money.

> Unfortunately when I think all the hurdles are solved; then we stumble over new ones.
> Yes unfortunately real world projects have definite requirements and budgets and deadlines.
> I do not see the point of open source software if it cannot solve commercial
> problems; unless Synapse is just an Academic Research project ? I am sure
> WS02 are using Synapse to solve real world commercial problems and not just
> for research. I am sure they make money from using the code.
>   
Of course, many real world users use Synapse - but some are shy to use 
the public forums, and risk exposing confidential information, security 
and passwords and sensitive data etc. I usually privately inform users 
whenever this happens.. since logs could reveal a lot of information 
which users sometimes are not aware of; and I remember I informed you 
too when you posted a Base64 encrypted password of a live server.

Synapse is not an academic project at all.. it never was. Multiple large 
health care providers in the US, the US Army, Governments in Asia and 
Europe, mobile resource management companies, one of the largest auto 
makers (who is still doing good in this economy), and SOA appliance 
vendor that embeds Synapse, European / Asian banks and a large media 
company across the US and Europe all use it. These are some of the users 
I know about.. and WSO2 would have more users too, that I am not aware of

Here are some "public" comments by a few users in the past:
"Thanks for the quick fix. Synapse continues to rock." 
<http://markmail.org/message/qbk4wvncjvtvat2s>
"Synapse continues to impress. Looking forward to 1.1 features upon its 
release" <http://markmail.org/message/zwyrzleaop25bqad>
"Thanks everyone for taking time to look into this -- you guys are great 
(know wonder Synapse is such a great product!)" 
<http://markmail.org/message/5lldrfesfb2wjv44>
"I have been using the excellent Apache Synapse lightweight Enterprise 
Service Bus over the last year or so, including on a live, production 
project." 
<http://thejavamonkey.blogspot.com/2008/09/apache-synapse-how-to-run-without.html> 

"We are using Apache Synapse version 1.1 to develop webservices for our 
mission critical "Surface Deployment and Distribution command" of U.S 
Army" <http://markmail.org/message/b67h6ggfh7vspt2u>

>"I do not see the point of open source software if it cannot solve commercial problems;"

I took this line out of the previous block, so that I can address it 
separately.. As far as I am aware, Open source software is not about 
building free software by volunteers to commercial companies who makes 
money out of it. For users its more about having lifetime access to the 
source code of the products, and knowing that licensing is free when you 
are ready to drop paying for support, so that you do not lock into 
proprietary vendor products, or with vendors who may not survive long 
term. When the code is available freely, you can compile and support it 
yourself, or get someone else to support it for you. In Apache, its 
about a community of people who are interested in building something.. 
In Apache we even consider 'Community over code', and enjoy working with 
diverse people and learning from them for us to improve. But Apache does 
not pay us! we are free to sell support or consulting to make sure we 
also survive :-)

Open source developers who volunteer their time, do not owe anything to 
the users - besides making sure the source code is available in full. 
Anyone who users the software is free to fix any issues, and if they 
like to contribute them back to the community.

> Unfortunately not all Synapse users are expert enough to contribute yet; so
> once again I mention the Catch 22 of  your comments. It takes time for new
> people to get up to speed so they can contribute.
>
> So I spent about 4 hours researching this today before posting; and I
> believe this is a contribution. We have real world requirements that would improve Synapse; this is a
> contribution.
>
> Asking questions that do not seem to be covered by documentation; so that
> others can benefit is a contribution. Thats all I can offer at the moment.
>   
I am fully aware that all first time users would not be using Synapse 
like they were pro's. From my past experience at WSO2, and at present, 
many users ask for help on a POC and/or training to get them started. 
They then develop an initial prototype project themselves and/or move 
into production with development support or professional services. 
Thereafter they use dev support to ask questions and learn more advanced 
stuff.

BTW, if I couldn't swim, I am aware there are many sites on the Internet 
that hosts loads of documentation, and even YouTube videos on how to 
swim all styles - free like open source software :-) . Now I may 
evaluate these techniques to see if they work in a pool of 5' depth.. 
but not in the open sea, not without knowing someone is there to help me 
if I get in trouble. I would rather pay a professional trainer to train 
me to swim, and then try it out more stuff when I am comfortable.. and 
of course blog about it later so that others would learn something from 
me too. When I can swim and know I can stay afloat, the YouTube videos 
would allow me to learn more cool stuff.. and maybe I could invent 
something new and contribute it back so that others could follow it too. 
I will not be like Adinnapubbaka in the story Mattakundali [3] and try 
to learn and do what I cannot.
> Unfortunately we have made an extensive investment in Synapse, in time, and
> at this point I am trying not to have to drop the use of this product and
> trying my hardest to demonstrate that this product can solve just one single
> real world problem for us. Am I in a hurry, yes, we have passed the deadline.
>   
So does this mean that you are trying to install Synapse into 
production?.. I would not invest much time or effort in an evaluation, 
but consider more options in parallel - maybe with some dev support to 
help me, to reduce my overall risk.

cheers
asankha

[1] http://edocs.bea.com/alsb/docs261/sftptransport/intro.html
[2] http://synapse.apache.org/Synapse_Samples.html#Sample254
[3] 
http://books.google.lk/books?id=sTclW3OWyw8C&pg=PA37&dq=%22mattakundali+(Flat-Earring)%22

-- 
Asankha C. Perera
AdroitLogic, http://adroitlogic.org

http://esbmagic.blogspot.com

Re: Can VFS SFTP be used to download todays files only

[Ruwan Linton <ru...@gmail.com>]

Kim, add to Asankha I must also emphasis that, you should try to learn on
getting something done from an open source community before starting to
learn Synapse and you multiple times stated that you have been using number
of open source software, is this how you have been interacted with those
communities as well?

I agree with Asankha and I think Synapse might not fit into your case, you
can either implement this by your own may be using synapse or may be with
yet another open source ESBs, there are may options around :-)

Thanks,
Ruwan

On Mon, Apr 6, 2009 at 3:50 PM, Asankha C. Perera <as...@apache.org>wrote:

> Hi Kim
>
>> 1) How do you specify a regex to download files with todays date that will
>> run in synapse unattended for weeks ?
>> Yes I can hard code an expression to match 31 March 2009 and after but
>> that
>> is no good today, is it. Java has the concept of Date NOW; this is what I
>> mean by today.
>> You say this is "Quite easy" please show me how, as I cannot see how this
>> can be done ? If it is easy then my problem is solved. Great, but this is
>> my
>> question ?
>>
>> 2)  If an SFTP server has 2000 old files permanently on it now that will
>> not
>> be removed; thats grows every day. How to I download just the new files ?
>> E.G Those placed on there with todays date in the name. I have no
>> permission
>> on the partners SFTP server to MOVE files or DELETE them, they just stay
>> there. It is not a good assumption that files can be MOVED or DELETED on
>> another organisations SFTP server. Very rarely in real B2B do you get this
>> luxury.
>>
>>
> The real world cases for which the transport has been originally written is
> to poll for a file or files at some specific location that matches some
> pattern, process them, and either 1) delete or 2) move them since the
> transport does not have state to remember what has been polled already.
>
> The commercial Aqualogic service bus follows this pattern too [1], while
> both Mule and Camel additionally supports the specification of date info in
> the file pattern. I couldn't readily find much about Fuse or JBoss etc.
>
> The solution I gave will not work, if you cannot move or delete processed
> files on the target sftp server, to prevent re-processing them later.
> However, supporting of date information in the pattern would be fairly easy
> to implement.
>
>> 3) You suggest SYnapse has a concept of new files ? How does this work ?
>>
>>
> This is if you are creating a new file.. e.g. saving a message received via
> http/s, JMS, email etc as a file - maybe to an sftp location like in [2]
>
>> 4) Are you saying that when I start Synapse with an SFTP mediator it
>> remembers every file downloaded ever in the past; so that when we re-start
>> Synapse (after maintenence) it remembers the old files. If this is true
>> this
>> is great. I am interested to know where this record of files is kept.
>> Where
>> is the file ?
>>
>> 5) Otherwise I am missing some very interesting behaviour of the Synapse
>> VFS
>> Mediator that does not seem to be provided by the samples or the
>> documentation or any prior Forum discussions.
>>
>>
> No, the transport is implemented as whats called an "Axis2 transport" ..
> its actually out of the Synapse core, and not directly dependent or linked.
> Implementing a record of processed files - even if possible would not be
> worth it - I think the use of Date in the regex is the most simple way to
> handle this
>
>> I am quite happy to raise a Jira but there is no need if Synapse can do
>> this, easily, as you say ?
>>
> Well.. the point of a JIRA is that all good ideas are captured by those who
> propose them, so that new developers or the current developers - when they
> have free time, can look at real world user suggestions for improvement and
> implement them for future versions
>
>> I do not understand this comment about being in a hurry; we are evaluating
>> Synapse to solve a commercial problem and unfortunately we have
>> encountered
>> many issues that we are trying to resolve.
>>
> I interpreted your multiple emails in the past about dropping or almost
> dropping the use of Synapse to mean that you are in a hurry to get this
> implemented in Synapse, and thats why I even suggested a pure Java
> implementation. Of course if you are still in evaluation, the best is to try
> to select another few ESBs (open source and/or commercial) and try to
> implement the same, and consider the total cost and the complexity of the
> final solution. In this case, consider the effort to implement whats missing
> in Synapse into the total cost of the solution.
>
> However, from some of your emails, I see production requirements being
> talked about, and thats why sometime back I suggested getting commercial
> support - in which case, someone else would have taken care of all of these
> issues - implementing what was missing as required. To be honest, you could
> have got a complete solution for your real world problem, with commercial
> support and a guarantee, probably for much less than the number of hours
> you've already spent and would spend * a reasonable hourly rate for the cost
> of your time, which again is money.
>
>  Unfortunately when I think all the hurdles are solved; then we stumble
>> over new ones.
>> Yes unfortunately real world projects have definite requirements and
>> budgets and deadlines.
>> I do not see the point of open source software if it cannot solve
>> commercial
>> problems; unless Synapse is just an Academic Research project ? I am sure
>> WS02 are using Synapse to solve real world commercial problems and not
>> just
>> for research. I am sure they make money from using the code.
>>
>>
> Of course, many real world users use Synapse - but some are shy to use the
> public forums, and risk exposing confidential information, security and
> passwords and sensitive data etc. I usually privately inform users whenever
> this happens.. since logs could reveal a lot of information which users
> sometimes are not aware of; and I remember I informed you too when you
> posted a Base64 encrypted password of a live server.
>
> Synapse is not an academic project at all.. it never was. Multiple large
> health care providers in the US, the US Army, Governments in Asia and
> Europe, mobile resource management companies, one of the largest auto makers
> (who is still doing good in this economy), and SOA appliance vendor that
> embeds Synapse, European / Asian banks and a large media company across the
> US and Europe all use it. These are some of the users I know about.. and
> WSO2 would have more users too, that I am not aware of
>
> Here are some "public" comments by a few users in the past:
> "Thanks for the quick fix. Synapse continues to rock." <
> http://markmail.org/message/qbk4wvncjvtvat2s>
> "Synapse continues to impress. Looking forward to 1.1 features upon its
> release" <http://markmail.org/message/zwyrzleaop25bqad>
> "Thanks everyone for taking time to look into this -- you guys are great
> (know wonder Synapse is such a great product!)" <
> http://markmail.org/message/5lldrfesfb2wjv44>
> "I have been using the excellent Apache Synapse lightweight Enterprise
> Service Bus over the last year or so, including on a live, production
> project." <
> http://thejavamonkey.blogspot.com/2008/09/apache-synapse-how-to-run-without.html>
>
> "We are using Apache Synapse version 1.1 to develop webservices for our
> mission critical "Surface Deployment and Distribution command" of U.S Army"
> <http://markmail.org/message/b67h6ggfh7vspt2u>
>
>  "I do not see the point of open source software if it cannot solve
>> commercial problems;"
>>
>
> I took this line out of the previous block, so that I can address it
> separately.. As far as I am aware, Open source software is not about
> building free software by volunteers to commercial companies who makes money
> out of it. For users its more about having lifetime access to the source
> code of the products, and knowing that licensing is free when you are ready
> to drop paying for support, so that you do not lock into proprietary vendor
> products, or with vendors who may not survive long term. When the code is
> available freely, you can compile and support it yourself, or get someone
> else to support it for you. In Apache, its about a community of people who
> are interested in building something.. In Apache we even consider 'Community
> over code', and enjoy working with diverse people and learning from them for
> us to improve. But Apache does not pay us! we are free to sell support or
> consulting to make sure we also survive :-)
>
> Open source developers who volunteer their time, do not owe anything to the
> users - besides making sure the source code is available in full. Anyone who
> users the software is free to fix any issues, and if they like to contribute
> them back to the community.
>
>  Unfortunately not all Synapse users are expert enough to contribute yet;
>> so
>> once again I mention the Catch 22 of  your comments. It takes time for new
>> people to get up to speed so they can contribute.
>>
>> So I spent about 4 hours researching this today before posting; and I
>> believe this is a contribution. We have real world requirements that would
>> improve Synapse; this is a
>> contribution.
>>
>> Asking questions that do not seem to be covered by documentation; so that
>> others can benefit is a contribution. Thats all I can offer at the moment.
>>
>>
> I am fully aware that all first time users would not be using Synapse like
> they were pro's. From my past experience at WSO2, and at present, many users
> ask for help on a POC and/or training to get them started. They then develop
> an initial prototype project themselves and/or move into production with
> development support or professional services. Thereafter they use dev
> support to ask questions and learn more advanced stuff.
>
> BTW, if I couldn't swim, I am aware there are many sites on the Internet
> that hosts loads of documentation, and even YouTube videos on how to swim
> all styles - free like open source software :-) . Now I may evaluate these
> techniques to see if they work in a pool of 5' depth.. but not in the open
> sea, not without knowing someone is there to help me if I get in trouble. I
> would rather pay a professional trainer to train me to swim, and then try it
> out more stuff when I am comfortable.. and of course blog about it later so
> that others would learn something from me too. When I can swim and know I
> can stay afloat, the YouTube videos would allow me to learn more cool
> stuff.. and maybe I could invent something new and contribute it back so
> that others could follow it too. I will not be like Adinnapubbaka in the
> story Mattakundali [3] and try to learn and do what I cannot.
>
>> Unfortunately we have made an extensive investment in Synapse, in time,
>> and
>> at this point I am trying not to have to drop the use of this product and
>> trying my hardest to demonstrate that this product can solve just one
>> single
>> real world problem for us. Am I in a hurry, yes, we have passed the
>> deadline.
>>
>>
> So does this mean that you are trying to install Synapse into production?..
> I would not invest much time or effort in an evaluation, but consider more
> options in parallel - maybe with some dev support to help me, to reduce my
> overall risk.
>
> cheers
> asankha
>
> [1] http://edocs.bea.com/alsb/docs261/sftptransport/intro.html
> [2] http://synapse.apache.org/Synapse_Samples.html#Sample254
> [3]
> http://books.google.lk/books?id=sTclW3OWyw8C&pg=PA37&dq=%22mattakundali+(Flat-Earring)%22<http://books.google.lk/books?id=sTclW3OWyw8C&pg=PA37&dq=%22mattakundali+%28Flat-Earring%29%22>
>
>
> --
> Asankha C. Perera
> AdroitLogic, http://adroitlogic.org
>
> http://esbmagic.blogspot.com
>
>
>
>
>


-- 
Ruwan Linton
Senior Software Engineer & Product Manager; WSO2 ESB; http://wso2.org/esb
WSO2 Inc.; http://wso2.org
email: ruwan@wso2.com; cell: +94 77 341 3097
blog: http://ruwansblog.blogspot.com

Re: Can VFS SFTP be used to download todays files only

["Asankha C. Perera" <as...@apache.org>]

Hi Kim
> 1) How do you specify a regex to download files with todays date that will
> run in synapse unattended for weeks ?
> Yes I can hard code an expression to match 31 March 2009 and after but that
> is no good today, is it. 
> Java has the concept of Date NOW; this is what I mean by today. 
>
> You say this is "Quite easy" please show me how, as I cannot see how this
> can be done ? If it is easy then my problem is solved. Great, but this is my
> question ?
>
> 2)  If an SFTP server has 2000 old files permanently on it now that will not
> be removed; thats grows every day. How to I download just the new files ?
> E.G Those placed on there with todays date in the name. I have no permission
> on the partners SFTP server to MOVE files or DELETE them, they just stay
> there. It is not a good assumption that files can be MOVED or DELETED on
> another organisations SFTP server. Very rarely in real B2B do you get this
> luxury.
>   
The real world cases for which the transport has been originally written 
is to poll for a file or files at some specific location that matches 
some pattern, process them, and either 1) delete or 2) move them since 
the transport does not have state to remember what has been polled already.

The commercial Aqualogic service bus follows this pattern too [1], while 
both Mule and Camel additionally supports the specification of date info 
in the file pattern. I couldn't readily find much about Fuse or JBoss etc.

The solution I gave will not work, if you cannot move or delete 
processed files on the target sftp server, to prevent re-processing them 
later. However, supporting of date information in the pattern would be 
fairly easy to implement.
> 3) You suggest SYnapse has a concept of new files ? How does this work ?
>   
This is if you are creating a new file.. e.g. saving a message received 
via http/s, JMS, email etc as a file - maybe to an sftp location like in [2]
> 4) Are you saying that when I start Synapse with an SFTP mediator it
> remembers every file downloaded ever in the past; so that when we re-start
> Synapse (after maintenence) it remembers the old files. If this is true this
> is great. I am interested to know where this record of files is kept. Where
> is the file ?
>
> 5) Otherwise I am missing some very interesting behaviour of the Synapse VFS
> Mediator that does not seem to be provided by the samples or the
> documentation or any prior Forum discussions.
>   
No, the transport is implemented as whats called an "Axis2 transport" .. 
its actually out of the Synapse core, and not directly dependent or 
linked. Implementing a record of processed files - even if possible 
would not be worth it - I think the use of Date in the regex is the most 
simple way to handle this
> I am quite happy to raise a Jira but there is no need if Synapse can do
> this, easily, as you say ? 
>   
Well.. the point of a JIRA is that all good ideas are captured by those 
who propose them, so that new developers or the current developers - 
when they have free time, can look at real world user suggestions for 
improvement and implement them for future versions
> I do not understand this comment about being in a hurry; we are evaluating
> Synapse to solve a commercial problem and unfortunately we have encountered
> many issues that we are trying to resolve.
I interpreted your multiple emails in the past about dropping or almost 
dropping the use of Synapse to mean that you are in a hurry to get this 
implemented in Synapse, and thats why I even suggested a pure Java 
implementation. Of course if you are still in evaluation, the best is to 
try to select another few ESBs (open source and/or commercial) and try 
to implement the same, and consider the total cost and the complexity of 
the final solution. In this case, consider the effort to implement whats 
missing in Synapse into the total cost of the solution.

However, from some of your emails, I see production requirements being 
talked about, and thats why sometime back I suggested getting commercial 
support - in which case, someone else would have taken care of all of 
these issues - implementing what was missing as required. To be honest, 
you could have got a complete solution for your real world problem, with 
commercial support and a guarantee, probably for much less than the 
number of hours you've already spent and would spend * a reasonable 
hourly rate for the cost of your time, which again is money.

> Unfortunately when I think all the hurdles are solved; then we stumble over new ones.
> Yes unfortunately real world projects have definite requirements and budgets and deadlines.
> I do not see the point of open source software if it cannot solve commercial
> problems; unless Synapse is just an Academic Research project ? I am sure
> WS02 are using Synapse to solve real world commercial problems and not just
> for research. I am sure they make money from using the code.
>   
Of course, many real world users use Synapse - but some are shy to use 
the public forums, and risk exposing confidential information, security 
and passwords and sensitive data etc. I usually privately inform users 
whenever this happens.. since logs could reveal a lot of information 
which users sometimes are not aware of; and I remember I informed you 
too when you posted a Base64 encrypted password of a live server.

Synapse is not an academic project at all.. it never was. Multiple large 
health care providers in the US, the US Army, Governments in Asia and 
Europe, mobile resource management companies, one of the largest auto 
makers (who is still doing good in this economy), and SOA appliance 
vendor that embeds Synapse, European / Asian banks and a large media 
company across the US and Europe all use it. These are some of the users 
I know about.. and WSO2 would have more users too, that I am not aware of

Here are some "public" comments by a few users in the past:
"Thanks for the quick fix. Synapse continues to rock." 
<http://markmail.org/message/qbk4wvncjvtvat2s>
"Synapse continues to impress. Looking forward to 1.1 features upon its 
release" <http://markmail.org/message/zwyrzleaop25bqad>
"Thanks everyone for taking time to look into this -- you guys are great 
(know wonder Synapse is such a great product!)" 
<http://markmail.org/message/5lldrfesfb2wjv44>
"I have been using the excellent Apache Synapse lightweight Enterprise 
Service Bus over the last year or so, including on a live, production 
project." 
<http://thejavamonkey.blogspot.com/2008/09/apache-synapse-how-to-run-without.html> 

"We are using Apache Synapse version 1.1 to develop webservices for our 
mission critical "Surface Deployment and Distribution command" of U.S 
Army" <http://markmail.org/message/b67h6ggfh7vspt2u>

>"I do not see the point of open source software if it cannot solve commercial problems;"

I took this line out of the previous block, so that I can address it 
separately.. As far as I am aware, Open source software is not about 
building free software by volunteers to commercial companies who makes 
money out of it. For users its more about having lifetime access to the 
source code of the products, and knowing that licensing is free when you 
are ready to drop paying for support, so that you do not lock into 
proprietary vendor products, or with vendors who may not survive long 
term. When the code is available freely, you can compile and support it 
yourself, or get someone else to support it for you. In Apache, its 
about a community of people who are interested in building something.. 
In Apache we even consider 'Community over code', and enjoy working with 
diverse people and learning from them for us to improve. But Apache does 
not pay us! we are free to sell support or consulting to make sure we 
also survive :-)

Open source developers who volunteer their time, do not owe anything to 
the users - besides making sure the source code is available in full. 
Anyone who users the software is free to fix any issues, and if they 
like to contribute them back to the community.

> Unfortunately not all Synapse users are expert enough to contribute yet; so
> once again I mention the Catch 22 of  your comments. It takes time for new
> people to get up to speed so they can contribute.
>
> So I spent about 4 hours researching this today before posting; and I
> believe this is a contribution. We have real world requirements that would improve Synapse; this is a
> contribution.
>
> Asking questions that do not seem to be covered by documentation; so that
> others can benefit is a contribution. Thats all I can offer at the moment.
>   
I am fully aware that all first time users would not be using Synapse 
like they were pro's. From my past experience at WSO2, and at present, 
many users ask for help on a POC and/or training to get them started. 
They then develop an initial prototype project themselves and/or move 
into production with development support or professional services. 
Thereafter they use dev support to ask questions and learn more advanced 
stuff.

BTW, if I couldn't swim, I am aware there are many sites on the Internet 
that hosts loads of documentation, and even YouTube videos on how to 
swim all styles - free like open source software :-) . Now I may 
evaluate these techniques to see if they work in a pool of 5' depth.. 
but not in the open sea, not without knowing someone is there to help me 
if I get in trouble. I would rather pay a professional trainer to train 
me to swim, and then try it out more stuff when I am comfortable.. and 
of course blog about it later so that others would learn something from 
me too. When I can swim and know I can stay afloat, the YouTube videos 
would allow me to learn more cool stuff.. and maybe I could invent 
something new and contribute it back so that others could follow it too. 
I will not be like Adinnapubbaka in the story Mattakundali [3] and try 
to learn and do what I cannot.
> Unfortunately we have made an extensive investment in Synapse, in time, and
> at this point I am trying not to have to drop the use of this product and
> trying my hardest to demonstrate that this product can solve just one single
> real world problem for us. Am I in a hurry, yes, we have passed the deadline.
>   
So does this mean that you are trying to install Synapse into 
production?.. I would not invest much time or effort in an evaluation, 
but consider more options in parallel - maybe with some dev support to 
help me, to reduce my overall risk.

cheers
asankha

[1] http://edocs.bea.com/alsb/docs261/sftptransport/intro.html
[2] http://synapse.apache.org/Synapse_Samples.html#Sample254
[3] 
http://books.google.lk/books?id=sTclW3OWyw8C&pg=PA37&dq=%22mattakundali+(Flat-Earring)%22

-- 
Asankha C. Perera
AdroitLogic, http://adroitlogic.org

http://esbmagic.blogspot.com

Re: Can VFS SFTP be used to download todays files only

[kimhorn <ki...@icsglobal.net>]

I am not sure you are correct in your responses below. The only proof that I
am wrong is providing the code to do this. 

1) How do you specify a regex to download files with todays date that will
run in synapse unattended for weeks ?
Yes I can hard code an expression to match 31 March 2009 and after but that
is no good today, is it. 
Java has the concept of Date NOW; this is what I mean by today. 

You say this is "Quite easy" please show me how, as I cannot see how this
can be done ? If it is easy then my problem is solved. Great, but this is my
question ?

2)  If an SFTP server has 2000 old files permanently on it now that will not
be removed; thats grows every day. How to I download just the new files ?
E.G Those placed on there with todays date in the name. I have no permission
on the partners SFTP server to MOVE files or DELETE them, they just stay
there. It is not a good assumption that files can be MOVED or DELETED on
another organisations SFTP server. Very rarely in real B2B do you get this
luxury.

3) You suggest SYnapse has a concept of new files ? How does this work ?

4) Are you saying that when I start Synapse with an SFTP mediator it
remembers every file downloaded ever in the past; so that when we re-start
Synapse (after maintenence) it remembers the old files. If this is true this
is great. I am interested to know where this record of files is kept. Where
is the file ?

5) Otherwise I am missing some very interesting behaviour of the Synapse VFS
Mediator that does not seem to be provided by the samples or the
documentation or any prior Forum discussions.


I am quite happy to raise a Jira but there is no need if Synapse can do
this, easily, as you say ?
But at this point only the code will help me understand this.


I do not understand this comment about being in a hurry; we are evaluating
Synapse to solve a commercial problem and unfortunately we have encountered
many issues that we are trying to resolve. 
We chose a particular set of use cases to solve and they were fixed months
ago. This is called an "architectural prototype". It is used to mitigate the
risk of a new product or type of solution/technology in a project.

Unfortunately when I think all the hurdles are solved; then we stumble over
new ones.
Yes unfortunately real world projects have definite requirements and budgets
and deadlines.
I do not see the point of open source software if it cannot solve commercial
problems; unless Synapse is just an Academic Research project ? I am sure
WS02 are using Synapse to solve real world commercial problems and not just
for research. I am sure they make money from using the code.

Unfortunately not all Synapse users are expert enough to contribute yet; so
once again I mention the Catch 22 of  your comments. It takes time for new
people to get up to speed so they can contribute.

So I spent about 4 hours researching this today before posting; and I
believe this is a contribution.
We have real world requirements that would improve Synapse; this is a
contribution.

Asking questions that do not seem to be covered by documentation; so that
others can benefit
is a contribution. Thats all I can offer at the moment.

Unfortunately we have made an extensive investment in Synapse, in time, and
at this point I am trying not to have to drop the use of this product and
trying my hardest to demonstrate that this product can solve just one single
real world problem for us. Am I in a hurry, yes, we have passed the
deadline.




Asankha C. Perera wrote:
> 
> Hi Kim
>> We need to download files created only today on a SFTP server. Initially
>> I
>> thought the regular expression parameter could be used to do this, but
>> now I
>> don't think its possible.
>>
>> A further major design fault of the VFS mediator is that I don't think
>> these
>> parameters can be programatically set.
> This is not a design fault, but the way it was designed.. To Synapse, 
> VFS is just a transport - underneath Axis2.. which makes things a bit 
> distant. Ofcourse it may not be exactly what you want, but has use for 
> some other common scenarios.. Synapse is expected to poll locations for 
> new files.. you can poll directories and also provide a file name pattern.
>> Requirement: The SFTP server we need to connect to keeps every file the
>> partner creates, on it for months. We only need files with todays date in
>> the name. Unfortunately we cannot change the partners policy here or hwo
>> they name files.
>>
>> The file naming scheme is:
>> CODEMMDDYYNN.TYPE
>> Where CODE is a set parameter
>> Where TYPE is a set parameter
>> Where NN is the increment number for files created within the day. So a *
>> here.
>> The rest is the date.
>>
>> So the server may have 2000 files and 3 created today.
>> How can I download just those 3 ?
>>
>> Originally I thought I could create the regular expresssion parameter in
>> Java code and pass to VFS.
>> But I don't think this is possible ?
>>   
> Not to poll, but you can do this to write - i.e. dynamically decide the 
> path to write. To poll, you should still be able to specify the regex 
> such that MMDDYY matches any dates after today, quite easily..
>> VFS parameter are isolated from Synapse. Never thought Synapse would be
>> so
>> inflexible. 
>>
>> Only solution left is to do it completely in a regular expression, not
>> sure
>> this is feasible ?
>> Can a Java regular expression know the date today ?
>> Is this compatible with how VFS uses the expression ?
>>
>> Is there any other way to set the VFS parameters ?
>>
>> Alternative is to fix the way VFS parameters are set, e.g via a property
>> rather than inflexible text only. (However, if VFS is a proxy this cannot
>> be
>> done either, because parameter cannot get set prior to proxy call)
>> OR
>> call VFS from Java - but then why use Synapse ?
>>
>> A final solution is to use a Java program to write the synapse.xml file,
>> with the required text. Then hot-load the synapse.xml every midnight.
>> However, again, may as well just stick with Java solution.
>>
>> This is about the last nail in the coffin for our problems with getting
>> Synapse to solve a simple mediation problem.
>>   
> Seems like using pure Java might be a good workaround for your case for 
> now, since you seem to be in a hurry, and all developers of Synapse are 
> volunteers and thus we cannot guarantee any time frames for 
> enhancements. We would be happy if you could open JIRA's though so that 
> someone could attend to them when they have time.
> 
> cheers
> asankha
> 
> -- 
> Asankha C. Perera
> AdroitLogic, http://adroitlogic.org
> 
> http://esbmagic.blogspot.com
> 
> 
> 
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Can-VFS-SFTP-be-used-to-download-todays-files-only-tp22901452p22903903.html
Sent from the Synapse - User mailing list archive at Nabble.com.

Re: Can VFS SFTP be used to download todays files only

["Asankha C. Perera" <as...@apache.org>]

Hi Kim
> We need to download files created only today on a SFTP server. Initially I
> thought the regular expression parameter could be used to do this, but now I
> don't think its possible.
>
> A further major design fault of the VFS mediator is that I don't think these
> parameters can be programatically set.
This is not a design fault, but the way it was designed.. To Synapse, 
VFS is just a transport - underneath Axis2.. which makes things a bit 
distant. Ofcourse it may not be exactly what you want, but has use for 
some other common scenarios.. Synapse is expected to poll locations for 
new files.. you can poll directories and also provide a file name pattern.
> Requirement: The SFTP server we need to connect to keeps every file the
> partner creates, on it for months. We only need files with todays date in
> the name. Unfortunately we cannot change the partners policy here or hwo
> they name files.
>
> The file naming scheme is:
> CODEMMDDYYNN.TYPE
> Where CODE is a set parameter
> Where TYPE is a set parameter
> Where NN is the increment number for files created within the day. So a *
> here.
> The rest is the date.
>
> So the server may have 2000 files and 3 created today.
> How can I download just those 3 ?
>
> Originally I thought I could create the regular expresssion parameter in
> Java code and pass to VFS.
> But I don't think this is possible ?
>   
Not to poll, but you can do this to write - i.e. dynamically decide the 
path to write. To poll, you should still be able to specify the regex 
such that MMDDYY matches any dates after today, quite easily..
> VFS parameter are isolated from Synapse. Never thought Synapse would be so
> inflexible. 
>
> Only solution left is to do it completely in a regular expression, not sure
> this is feasible ?
> Can a Java regular expression know the date today ?
> Is this compatible with how VFS uses the expression ?
>
> Is there any other way to set the VFS parameters ?
>
> Alternative is to fix the way VFS parameters are set, e.g via a property
> rather than inflexible text only. (However, if VFS is a proxy this cannot be
> done either, because parameter cannot get set prior to proxy call)
> OR
> call VFS from Java - but then why use Synapse ?
>
> A final solution is to use a Java program to write the synapse.xml file,
> with the required text. Then hot-load the synapse.xml every midnight.
> However, again, may as well just stick with Java solution.
>
> This is about the last nail in the coffin for our problems with getting
> Synapse to solve a simple mediation problem.
>   
Seems like using pure Java might be a good workaround for your case for 
now, since you seem to be in a hurry, and all developers of Synapse are 
volunteers and thus we cannot guarantee any time frames for 
enhancements. We would be happy if you could open JIRA's though so that 
someone could attend to them when they have time.

cheers
asankha

-- 
Asankha C. Perera
AdroitLogic, http://adroitlogic.org

http://esbmagic.blogspot.com