You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Khawaja Shams <ks...@gmail.com> on 2005/12/27 02:20:18 UTC
detecting a new file
Hello all,
I am curious if anyone has ever had to write a servlet that listens for
new files on the server and processes them upon appearence. Basically, I
need to write a server side program that detects when (our automated
process) delivers a file to the server filesystem and then process the
file. I would sincerely appreciate any pointers on how this can be
accomplished from within tomcat.
Happy Holidays.
Sincerely,
Khawaja Shams
Re: detecting a new file
Posted by Jeremy Thomerson <co...@jeremythomerson.com>.
If you still need a general scheduling solution, or the suggestion(s) below
won't work, perhaps you could investigate using Quartz and configuring it
within your
container (Spring is handy for this).
http://www.opensymphony.com/quartz/
Article on others using it:
http://www.experts-exchange.com/Programming/Programming_Languages/Java/Q_21288885.html
(scroll to bottom)
I've used Quartz / Spring very successfully for scheduled (cron) jobs. It
takes cron-formatted strings for scheduling. I haven't used it within the
Servlet container, but I'd imagine it should work just as well.
Jeremy Thomerson
----- Original Message -----
From: "Wade Chandler" <hw...@yahoo.com>
To: "Tomcat Users List" <us...@tomcat.apache.org>
Sent: Monday, December 26, 2005 9:15 PM
Subject: Re: detecting a new file
> --- Khawaja Shams <ks...@gmail.com> wrote:
>
>> Hello all,
>> I am curious if anyone has ever had to write a
>> servlet that listens for
>> new files on the server and processes them upon
>> appearence. Basically, I
>> need to write a server side program that detects
>> when (our automated
>> process) delivers a file to the server filesystem
>> and then process the
>> file. I would sincerely appreciate any pointers on
>> how this can be
>> accomplished from within tomcat.
>>
>>
>> Happy Holidays.
>>
>>
>> Sincerely,
>> Khawaja Shams
>>
>
>
> My theory is always copy something that is working
> now. Go to www.netbeans.org,download a source zip as
> available here
> (http://www.netbeans.info/downloads/download.php?a=n&p=2)
> ... maybe choose a daily build ... keep following the
> arrows eventually you'll get to the download and be
> able to select a source download...that or figure out
> how to use their CVS repository, and then follow the
> source directory structure to the directory (relative
> obviously):
> openide\fs\src\org\openide\filesystems
>
> see the files:
> FileChangeListener.java
> FileChangeAdapter.java
> FileEvent.java
> AbstractFolder.java (this actually has the code to
> watch the directory)
>
> also the tomcat code watches directory contents and
> files as well. I don't know exactly where, but you'll
> be looking for code that watches the webapps directory
> for .war files and possibly other files. Auto-deploy
> of war files works this way.
>
> Anyways, all you're doing is watching a directory for
> files and storing previous states so you can compare
> later to see what has happened. You could store state
> in a DB and use SQL to determine changes or store it
> in memory. This will all depend on what you need ...
> speed vs. memory usage. SQL and connections obviously
> slower than an in memory map, but it should save you
> on always keeping your memory used.
>
> Hope it helps,
>
> Wade
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: detecting a new file
Posted by Wade Chandler <hw...@yahoo.com>.
--- Khawaja Shams <ks...@gmail.com> wrote:
> Hello all,
> I am curious if anyone has ever had to write a
> servlet that listens for
> new files on the server and processes them upon
> appearence. Basically, I
> need to write a server side program that detects
> when (our automated
> process) delivers a file to the server filesystem
> and then process the
> file. I would sincerely appreciate any pointers on
> how this can be
> accomplished from within tomcat.
>
>
> Happy Holidays.
>
>
> Sincerely,
> Khawaja Shams
>
My theory is always copy something that is working
now. Go to www.netbeans.org,download a source zip as
available here
(http://www.netbeans.info/downloads/download.php?a=n&p=2)
... maybe choose a daily build ... keep following the
arrows eventually you'll get to the download and be
able to select a source download...that or figure out
how to use their CVS repository, and then follow the
source directory structure to the directory (relative
obviously):
openide\fs\src\org\openide\filesystems
see the files:
FileChangeListener.java
FileChangeAdapter.java
FileEvent.java
AbstractFolder.java (this actually has the code to
watch the directory)
also the tomcat code watches directory contents and
files as well. I don't know exactly where, but you'll
be looking for code that watches the webapps directory
for .war files and possibly other files. Auto-deploy
of war files works this way.
Anyways, all you're doing is watching a directory for
files and storing previous states so you can compare
later to see what has happened. You could store state
in a DB and use SQL to determine changes or store it
in memory. This will all depend on what you need ...
speed vs. memory usage. SQL and connections obviously
slower than an in memory map, but it should save you
on always keeping your memory used.
Hope it helps,
Wade
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: detecting a new file
Posted by Wade Chandler <hw...@yahoo.com>.
--- Daniel Blumenthal <da...@wordchamp.com> wrote:
> A related question to this is, how can you set up a
> process within the
> servlet that fires off at X time, or every Y
> seconds? The equivalent of a
> cron job inside the servlet.
>
> If you have high enough traffic, there's an easy
> hack - just store a Date
> object as a servlet-level attribute, and check on
> each request if the
> current time is greater than or equal. But is there
> a cleaner solution?
> Also, it would be nice to have a little more precise
> control than hoping
> that a user will happen to come to the site at the
> right time.
>
> I suppose you could spin off a thread that sleeps
> for X amount of time,
> wakes to check on the situation, and goes back to
> sleep when done.
>
> To the specific question - I would guess
> (incorrectly?) that there's no Java
> file system listener. But I could certainly be
> wrong... ?
>
> Daniel
>
> > -----Original Message-----
> > From: Khawaja Shams [mailto:ksshams@gmail.com]
> > Sent: Monday, December 26, 2005 8:20 PM
> > To: Tomcat Users List
> > Subject: detecting a new file
> >
> > Hello all,
> > I am curious if anyone has ever had to write a
> servlet
> > that listens for new files on the server and
> processes them
> > upon appearence. Basically, I need to write a
> server side
> > program that detects when (our automated
> > process) delivers a file to the server filesystem
> and then
> > process the file. I would sincerely appreciate
> any pointers
> > on how this can be accomplished from within
> tomcat.
> >
> >
> > Happy Holidays.
> >
> >
> > Sincerely,
> > Khawaja Shams
> >
There is no standard file system watcher...at least
not in standard API's which I know of. Yes, you have
to use a thread or java.util.Timer (uses a thread
behind the abstraction and might make it easier for
you) to do this unless you use SharedMemory or TCP/IP
and spawn off a separate process so you don't break
the rule about not creating threads in the container
(but I don't want to complicate this discussion), but
I prefer to go by what I need when I need it vs the
spec ( ;-) ) utility threads like this are needed at
times depending on what you need/have to accomplish.
I would use a thread for this as it's pretty simple
stuff. Just make sure you protect the tread with good
try-catch so you don't bust out of the watching thread
on some goofy error...you'll have to weigh this
obviously. Setup a simple event mechanism like:
public interface FileChangeListener extends
java.util.EventListener {
public void fileChanged(FileChangeEvent evt);
}
public class FileChangeEvent extends
java.util.EventObject {
public static final int TYPE_NEW = 1;
public static final int TYPE_MODIFIED = 2;
private int eventType = TYPE_NEW;
private MonitoredFile source = null;
private MonitoredFile watched = null;
public FileChangeEvent(){
}
public void setSource(MonitoredFile source){
this.source = source;
}
public MonitoredFile getSource(){
return this.source;
}
//.......
//put other setters here for watched and type
}
public class MonitoredFile extends File {
public void
addFileChangeListener(FileChangeListener listener){
//if using a JDK/JRE with
javax.swing.event.EventListenerList
then use that for holding the events...just makes it
a little easier
}
public void
removeFileChangeListener(FileChangeListener listener){
//use your javax.swing.event.EventListenerList
}
public FileChangeListener[]
getFileChangeListeners(){
//use your javax.swing.event.EventListenerList
}
public void fireFileChangeEvent(FileChangeEvent
evt) throws Throwable {
//get all the listeners set for FileChangeEvent
and call fileChanged(evt)
}
}
....
now all you would have to do is to create a class for
watching your directory which you can do different
things...either watch a directory and all sub
directories....if you don't watch sub directories you
can simply call File.listFiles or File.list depending
on whether you want File objects or String objects.
If you want to watch all the directories then you can
use this code I'll give you ... what the heck.....
public static void enumerateFiles(Vector out,
String dir, int howDeep) throws Throwable {
enumerateFiles(out, dir, 0, howDeep);
}
/**
*File used to recursively run over and sort a
directory hierarchy to flatten out the directory
*structure of a file system into an array or in
this case a Vector.
*@param out the Vector the enumerated file names
will be written to. These will be sorted with each
* directory listing with
java.util.Arrays.sort(File[]) so see that method for
the format of the
* sort
*@param dir the directory we want the listing for
*@param deepCounter the counter for tracking how
deep we can go. This should be 0 in normal calls
*@param howDeep the number of directory levels
deep we want to return
*/
protected static void
enumerateFiles(java.util.Vector out, String dir, int
deepCounter, int howDeep) throws Throwable {
deepCounter++;
if(howDeep>=0&&deepCounter>howDeep){
return;
}
java.io.File f = new java.io.File(dir);
if(!f.isDirectory()){
throw new IllegalArgumentException("The
given directory "+dir+" is not a valid directory.");
}
java.io.File[] files = f.listFiles();
java.util.Arrays.sort(files);
for(int i = 0;
files!=null&&i<files.length;i++){
out.add(files[i].getAbsoluteFile().getCanonicalPath());
if(files[i].isDirectory()){
enumerateFiles(out,
files[i].getAbsoluteFile().getCanonicalPath(),
(deepCounter), howDeep);
}
}
}
Run that code and you'll see how it organizes the
returned files. Works really well, and you can even
control how deep you want to expose the directory
structure....moving on
So, all you have to do now is write your thread and
decide on how you want to fire events on your new
MonitoredFile objects...fire events on new files and
fire events on files that change and possibly
directories that change....what ever you want. You
can easily tell if something changes or is added by
storing off a set of info in a RandomAccessFile say
first column is 1000 (just to be safe) null terminate
it and is the file name and full path, second a
creation date and time (as long), third a modification
date and time (as long), and file size (as long) You
could cram the long dates into a long and actually
make the date and time show up within the long and
only take up 8 bytes instead of more i.e.
20050101010101111 which is YYYYMMDDHHMMSSsss where
s=millisecond space holder. and then 3 bytes for the
time zone if needed, but for this it's probably
not...get the dates to and from longs with:
public static final long
convertLongDateToReadableLongDate(long date)
{
long ret = 0;
GregorianCalendar cal = new GregorianCalendar();
cal.setTimeInMillis(date);
long year = cal.get(cal.YEAR);
long month = cal.get(cal.MONTH) + 1;
long day = cal.get(cal.DAY_OF_MONTH);
long hour = cal.get(cal.HOUR_OF_DAY);
long minute = cal.get(cal.MINUTE);
long sec = cal.get(cal.SECOND);
long millis = cal.get(cal.MILLISECOND);
ret = (year * 10000000000000L);
ret += (month * 100000000000L);
ret += (day * 1000000000L);
ret += (hour * 10000000L);
ret += (minute * 100000L);
ret += (sec * 1000L);
ret += millis;
return ret;
}
public static final long
convertPackedLongDateToLongDate(long date)
{
if( date == PDBase.NULL_DATE )
{
return date;
}
long ret = 0;
String sdate = ""+date;
if( sdate.length() != 17 )
{
throw new IllegalArgumentException("The date
is not a real packed date.");
}
char[] cyear = new char[4];
char[] cmonth = new char[2];
char[] cday = new char[2];
char[] chh = new char[2];
char[] cmm = new char[2];
char[] css = new char[2];
char[] cmmm = new char[3];
cyear[0] = sdate.charAt(0);
cyear[1] = sdate.charAt(1);
cyear[2] = sdate.charAt(2);
cyear[3] = sdate.charAt(3);
cmonth[0] = sdate.charAt(4);
cmonth[1] = sdate.charAt(5);
cday[0] = sdate.charAt(6);
cday[1] = sdate.charAt(7);
chh[0] = sdate.charAt(8);
chh[1] = sdate.charAt(9);
cmm[0] = sdate.charAt(10);
cmm[1] = sdate.charAt(11);
css[0] = sdate.charAt(12);
css[1] = sdate.charAt(13);
cmmm[0] = sdate.charAt(14);
cmmm[1] = sdate.charAt(15);
cmmm[2] = sdate.charAt(16);
int year = Integer.parseInt(new String(cyear));
int month = Integer.parseInt(new String(cmonth))
- 1;
int day = Integer.parseInt(new String(cday));
int hh = Integer.parseInt(new String(chh));
int mm = Integer.parseInt(new String(cmm));
int ss = Integer.parseInt(new String(css));
int mmm = Integer.parseInt(new String(cmmm));
GregorianCalendar cal = new GregorianCalendar();
cal.set(cal.YEAR, year);
cal.set(cal.MONTH, month);
cal.set(cal.DAY_OF_MONTH, day);
cal.set(cal.HOUR_OF_DAY, hh);
cal.set(cal.MINUTE, mm);
cal.set(cal.SECOND, ss);
cal.set(cal.MILLISECOND, mmm);
ret = cal.getTimeInMillis();
return ret;
}
that'll make it readable if you store this stuff in a
RandomAccessFile or DB and need to look at it later.
Now all you need to do is wake up your thread every so
often and do some checks. I'll leave the part of
figuring out how to use the information you'll have
available, writing it and reading it to and from the
RandomAccessFile to you...shouldn't be that difficult
though. There are a couple of different schemes you
could use. You might not even store off this stuff
and instead keep MonitoredFile objects for every
single file name you get back from enumerating the
files. Then what you need to do is check all of the
MonitoredFile Objects you have every time to see if
they are deleted or still exist or if any of their
other information has changed or if you have any new
ones. You could store them off in map by file name so
then you're able to see if they existed before or not.
You could even store them all out in .ser files
(meaning serialize the objects) and then just store
the serialized file names in a simple text file...line
by line and read them back in this way when you wake
up to perform some checks the scheme used should be
dictated by how often you think you should wake up.
Obviously loading that many files could take some
time, so you'll have to figure out what suits your
needs.
Some of this may be jumbled up and some pieces
missing, but hey it's on the fly and should get you
started. You might even rip some TC code or Netbeans
or Eclipse code which does this type of stuff. For a
starter I might use java.util.Timer and create start
and stop and destroy this guy in my
ServletContextListener webapp comes up start goes down
off.
Hope it helps,
Wade
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: detecting a new file
Posted by Daniel Blumenthal <da...@wordchamp.com>.
A related question to this is, how can you set up a process within the
servlet that fires off at X time, or every Y seconds? The equivalent of a
cron job inside the servlet.
If you have high enough traffic, there's an easy hack - just store a Date
object as a servlet-level attribute, and check on each request if the
current time is greater than or equal. But is there a cleaner solution?
Also, it would be nice to have a little more precise control than hoping
that a user will happen to come to the site at the right time.
I suppose you could spin off a thread that sleeps for X amount of time,
wakes to check on the situation, and goes back to sleep when done.
To the specific question - I would guess (incorrectly?) that there's no Java
file system listener. But I could certainly be wrong... ?
Daniel
> -----Original Message-----
> From: Khawaja Shams [mailto:ksshams@gmail.com]
> Sent: Monday, December 26, 2005 8:20 PM
> To: Tomcat Users List
> Subject: detecting a new file
>
> Hello all,
> I am curious if anyone has ever had to write a servlet
> that listens for new files on the server and processes them
> upon appearence. Basically, I need to write a server side
> program that detects when (our automated
> process) delivers a file to the server filesystem and then
> process the file. I would sincerely appreciate any pointers
> on how this can be accomplished from within tomcat.
>
>
> Happy Holidays.
>
>
> Sincerely,
> Khawaja Shams
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org