You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-user@db.apache.org by Ryan P Bobko <ry...@ostrich-emulators.com> on 2006/03/23 00:19:55 UTC
advice for client/server application
Hi List,
First of all, I can't say enough how impressed I've been with Derby. Every
time I've thought this embedded wouldn't be able to do something I expect
from a "full-blown" database (nested selects, correlated subqueries, stored
procedures, you name it), it's suprised me. I love it.
This isn't strictly a Derby question, but I'm hoping for some advice or
suggestions with how to procede. I've been working on an application that is
a sort of half-database, half-FTP client/server setup. The protocol I've
implemented between the client and server lets the app do things like run
queries, but also move files around based on those results. Or insert rows
into the database based on where files have moved to. Files can be moved from
the server to client and vice versa.
Things have been working just fine, except that when I run queries, the server
process does all the work and returns the results as a vector of string
arrays. It's never sat well with me--and as you can imagine--now that the
dataset is getting pretty big (120.000-4KB rows returned for some queries),
I'm using too much memory.
What I'd like to do is get my jdbc connection object onto the client so I
don't have to "package" everything up when returning resultsets. The question
is how? My first idea was to just use derby's network server and write the
file protocol separately, but I'd prefer to stick with just one socket if I
can.
Advice? Thanks for your time.
ry
--
Technological progress has merely provided us with more efficient means
for going backwards.
--Aldous Huxley
Re: advice for client/server application
Posted by Veaceslav Chicu <cs...@infologic.fr>.
java.sql.Statement
public void setFetchSize(int rows)
throws SQLException
Gives the JDBC driver a hint as to the number of rows that should be
fetched from the database when more rows are needed for this ResultSet
object. If the fetch size specified is zero, the JDBC driver ignores the
value and is free to make its own best guess as to what the fetch size
should be. The default value is set by the Statement object that created
the result set. The fetch size may be changed at any time.
Parameters:
rows - the number of rows to fetch
Throws:
SQLException - if a database access error occurs or the
condition 0 <= rows <= this.getMaxRows() is not satisfied
Since:
1.2
best regards,
Slavic
David W. Van Couvering wrote:
> A common way client applications working with large result sets have
> handled the "too much memory" problem that I've seen is to send the
> results over in chunks. Instead of sending all 120,000 records in one
> response, just send 100 or 1,000. The client processes those 1,000
> records, throws them away, and get the next 1,000.
>
> Would that work for you?
>
> David
>
> Ryan P Bobko wrote:
>> Hi List,
>> First of all, I can't say enough how impressed I've been with Derby.
>> Every time I've thought this embedded wouldn't be able to do something
>> I expect from a "full-blown" database (nested selects, correlated
>> subqueries, stored procedures, you name it), it's suprised me. I love it.
>>
>> This isn't strictly a Derby question, but I'm hoping for some advice
>> or suggestions with how to procede. I've been working on an
>> application that is a sort of half-database, half-FTP client/server
>> setup. The protocol I've implemented between the client and server
>> lets the app do things like run queries, but also move files around
>> based on those results. Or insert rows into the database based on
>> where files have moved to. Files can be moved from the server to
>> client and vice versa.
>>
>> Things have been working just fine, except that when I run queries,
>> the server process does all the work and returns the results as a
>> vector of string arrays. It's never sat well with me--and as you can
>> imagine--now that the dataset is getting pretty big (120.000-4KB rows
>> returned for some queries), I'm using too much memory.
>>
>> What I'd like to do is get my jdbc connection object onto the client
>> so I don't have to "package" everything up when returning resultsets.
>> The question is how? My first idea was to just use derby's network
>> server and write the file protocol separately, but I'd prefer to stick
>> with just one socket if I can.
>> Advice? Thanks for your time.
>> ry
>>
Re: advice for client/server application
Posted by Ryan P Bobko <ry...@ostrich-emulators.com>.
Oh, the usual. Stick your head in the ground or run away when anything
happens. Maybe it doesn't translate well? Darn this English as a first
language!
On Thursday 23 March 2006 07:02 pm, David W. Van Couvering wrote:
> I do have to ask, what does it mean to emulate an ostrich?
>
> David
>
> Ryan P Bobko wrote:
> > Thanks for the advice. That's probably what I needed to hear.
> >
> > ry
> >
> > On Thursday 23 March 2006 06:31 pm, David W. Van Couvering wrote:
> >>Yes, I see what you're saying, you're rewriting JDBC extended with file
> >>transfer.
> >>
> >>You could take the Derby source and extend it so that you can do file
> >>transfer functionality, but then you're stuck with your own fork of
> >>JDBC. And I'm sorry to say that the Derby community would likely not
> >>accept your changes back (someone correct me if I'm wrong) because there
> >>would be non-standard extensions to JDBC, and we strive to be
> >>standards-compliant as a community.
> >>
> >>Personally, I would just open two connections: one for JDBC work and one
> >>for file transfer. Keep them separate.
> >>
> >>David
> >>
> >>Ryan P Bobko wrote:
> >>>Thanks for the advice, but I'm not sure if that will help. I now realize
> >>>what makes me uneasy about my architecture is that I feel like I'm
> >>>rewriting JDBC little by little just so I can have my file-moving piece
> >>>on top of it. JDBC works great for my purposes, so my preference would
> >>> be to remove my custom protocol whenever possible. Is it feasible to
> >>> extend the JDBC API? Or will that just be more trouble than it's worth?
> >>>
> >>>I realize I just changed my question. Any more advice?
> >>>
> >>>ry
> >>>
> >>>On Wednesday 22 March 2006 07:23 pm, David W. Van Couvering wrote:
> >>>>A common way client applications working with large result sets have
> >>>>handled the "too much memory" problem that I've seen is to send the
> >>>>results over in chunks. Instead of sending all 120,000 records in one
> >>>>response, just send 100 or 1,000. The client processes those 1,000
> >>>>records, throws them away, and get the next 1,000.
> >>>>
> >>>>Would that work for you?
> >>>>
> >>>>David
> >>>>
> >>>>Ryan P Bobko wrote:
> >>>>>Hi List,
> >>>>>First of all, I can't say enough how impressed I've been with Derby.
> >>>>>Every time I've thought this embedded wouldn't be able to do something
> >>>>> I expect from a "full-blown" database (nested selects, correlated
> >>>>> subqueries, stored procedures, you name it), it's suprised me. I love
> >>>>> it.
> >>>>>
> >>>>>This isn't strictly a Derby question, but I'm hoping for some advice
> >>>>> or suggestions with how to procede. I've been working on an
> >>>>> application that is a sort of half-database, half-FTP client/server
> >>>>> setup. The protocol I've implemented between the client and server
> >>>>> lets the app do things like run queries, but also move files around
> >>>>> based on those results. Or insert rows into the database based on
> >>>>> where files have moved to. Files can be moved from the server to
> >>>>> client and vice versa.
> >>>>>
> >>>>>Things have been working just fine, except that when I run queries,
> >>>>> the server process does all the work and returns the results as a
> >>>>> vector of string arrays. It's never sat well with me--and as you can
> >>>>> imagine--now that the dataset is getting pretty big (120.000-4KB rows
> >>>>> returned for some queries), I'm using too much memory.
> >>>>>
> >>>>>What I'd like to do is get my jdbc connection object onto the client
> >>>>> so I don't have to "package" everything up when returning resultsets.
> >>>>> The question is how? My first idea was to just use derby's network
> >>>>> server and write the file protocol separately, but I'd prefer to
> >>>>> stick with just one socket if I can.
> >>>>>
> >>>>>Advice? Thanks for your time.
> >>>>>ry
--
When I turn into one of them, I will no longer be Jacob.
I will be a lapdog of Satan.
Re: advice for client/server application
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
I do have to ask, what does it mean to emulate an ostrich?
David
Ryan P Bobko wrote:
> Thanks for the advice. That's probably what I needed to hear.
>
> ry
>
> On Thursday 23 March 2006 06:31 pm, David W. Van Couvering wrote:
>
>>Yes, I see what you're saying, you're rewriting JDBC extended with file
>>transfer.
>>
>>You could take the Derby source and extend it so that you can do file
>>transfer functionality, but then you're stuck with your own fork of
>>JDBC. And I'm sorry to say that the Derby community would likely not
>>accept your changes back (someone correct me if I'm wrong) because there
>>would be non-standard extensions to JDBC, and we strive to be
>>standards-compliant as a community.
>>
>>Personally, I would just open two connections: one for JDBC work and one
>>for file transfer. Keep them separate.
>>
>>David
>>
>>Ryan P Bobko wrote:
>>
>>>Thanks for the advice, but I'm not sure if that will help. I now realize
>>>what makes me uneasy about my architecture is that I feel like I'm
>>>rewriting JDBC little by little just so I can have my file-moving piece
>>>on top of it. JDBC works great for my purposes, so my preference would be
>>>to remove my custom protocol whenever possible. Is it feasible to extend
>>>the JDBC API? Or will that just be more trouble than it's worth?
>>>
>>>I realize I just changed my question. Any more advice?
>>>
>>>ry
>>>
>>>On Wednesday 22 March 2006 07:23 pm, David W. Van Couvering wrote:
>>>
>>>>A common way client applications working with large result sets have
>>>>handled the "too much memory" problem that I've seen is to send the
>>>>results over in chunks. Instead of sending all 120,000 records in one
>>>>response, just send 100 or 1,000. The client processes those 1,000
>>>>records, throws them away, and get the next 1,000.
>>>>
>>>>Would that work for you?
>>>>
>>>>David
>>>>
>>>>Ryan P Bobko wrote:
>>>>
>>>>>Hi List,
>>>>>First of all, I can't say enough how impressed I've been with Derby.
>>>>>Every time I've thought this embedded wouldn't be able to do something I
>>>>>expect from a "full-blown" database (nested selects, correlated
>>>>>subqueries, stored procedures, you name it), it's suprised me. I love
>>>>>it.
>>>>>
>>>>>This isn't strictly a Derby question, but I'm hoping for some advice or
>>>>>suggestions with how to procede. I've been working on an application
>>>>>that is a sort of half-database, half-FTP client/server setup. The
>>>>>protocol I've implemented between the client and server lets the app do
>>>>>things like run queries, but also move files around based on those
>>>>>results. Or insert rows into the database based on where files have
>>>>>moved to. Files can be moved from the server to client and vice versa.
>>>>>
>>>>>Things have been working just fine, except that when I run queries, the
>>>>>server process does all the work and returns the results as a vector of
>>>>>string arrays. It's never sat well with me--and as you can imagine--now
>>>>>that the dataset is getting pretty big (120.000-4KB rows returned for
>>>>>some queries), I'm using too much memory.
>>>>>
>>>>>What I'd like to do is get my jdbc connection object onto the client so
>>>>>I don't have to "package" everything up when returning resultsets. The
>>>>>question is how? My first idea was to just use derby's network server
>>>>>and write the file protocol separately, but I'd prefer to stick with
>>>>>just one socket if I can.
>>>>>
>>>>>Advice? Thanks for your time.
>>>>>ry
>
>
Re: advice for client/server application
Posted by Ryan P Bobko <ry...@ostrich-emulators.com>.
Thanks for the advice. That's probably what I needed to hear.
ry
On Thursday 23 March 2006 06:31 pm, David W. Van Couvering wrote:
> Yes, I see what you're saying, you're rewriting JDBC extended with file
> transfer.
>
> You could take the Derby source and extend it so that you can do file
> transfer functionality, but then you're stuck with your own fork of
> JDBC. And I'm sorry to say that the Derby community would likely not
> accept your changes back (someone correct me if I'm wrong) because there
> would be non-standard extensions to JDBC, and we strive to be
> standards-compliant as a community.
>
> Personally, I would just open two connections: one for JDBC work and one
> for file transfer. Keep them separate.
>
> David
>
> Ryan P Bobko wrote:
> > Thanks for the advice, but I'm not sure if that will help. I now realize
> > what makes me uneasy about my architecture is that I feel like I'm
> > rewriting JDBC little by little just so I can have my file-moving piece
> > on top of it. JDBC works great for my purposes, so my preference would be
> > to remove my custom protocol whenever possible. Is it feasible to extend
> > the JDBC API? Or will that just be more trouble than it's worth?
> >
> > I realize I just changed my question. Any more advice?
> >
> > ry
> >
> > On Wednesday 22 March 2006 07:23 pm, David W. Van Couvering wrote:
> >>A common way client applications working with large result sets have
> >>handled the "too much memory" problem that I've seen is to send the
> >>results over in chunks. Instead of sending all 120,000 records in one
> >>response, just send 100 or 1,000. The client processes those 1,000
> >>records, throws them away, and get the next 1,000.
> >>
> >>Would that work for you?
> >>
> >>David
> >>
> >>Ryan P Bobko wrote:
> >>>Hi List,
> >>>First of all, I can't say enough how impressed I've been with Derby.
> >>>Every time I've thought this embedded wouldn't be able to do something I
> >>>expect from a "full-blown" database (nested selects, correlated
> >>>subqueries, stored procedures, you name it), it's suprised me. I love
> >>> it.
> >>>
> >>>This isn't strictly a Derby question, but I'm hoping for some advice or
> >>>suggestions with how to procede. I've been working on an application
> >>> that is a sort of half-database, half-FTP client/server setup. The
> >>> protocol I've implemented between the client and server lets the app do
> >>> things like run queries, but also move files around based on those
> >>> results. Or insert rows into the database based on where files have
> >>> moved to. Files can be moved from the server to client and vice versa.
> >>>
> >>>Things have been working just fine, except that when I run queries, the
> >>>server process does all the work and returns the results as a vector of
> >>>string arrays. It's never sat well with me--and as you can imagine--now
> >>>that the dataset is getting pretty big (120.000-4KB rows returned for
> >>>some queries), I'm using too much memory.
> >>>
> >>>What I'd like to do is get my jdbc connection object onto the client so
> >>> I don't have to "package" everything up when returning resultsets. The
> >>> question is how? My first idea was to just use derby's network server
> >>> and write the file protocol separately, but I'd prefer to stick with
> >>> just one socket if I can.
> >>>
> >>>Advice? Thanks for your time.
> >>>ry
--
There are three ways to get something done: do it yourself, hire
someone, or forbid your kids to do it.
Re: advice for client/server application
Posted by Daniel John Debrunner <dj...@apache.org>.
David W. Van Couvering wrote:
> Yes, I see what you're saying, you're rewriting JDBC extended with file
> transfer.
>
> You could take the Derby source and extend it so that you can do file
> transfer functionality, but then you're stuck with your own fork of
> JDBC. And I'm sorry to say that the Derby community would likely not
> accept your changes back (someone correct me if I'm wrong) because there
> would be non-standard extensions to JDBC, and we strive to be
> standards-compliant as a community.
I'm not so sure, some pluggable api that would allow additional
communication over the same tcp/ip connection seems like an interesting
direction. Innovation around open source. I think Oracle has some way of
passing async events back to the client over the JDBC connection.
> Personally, I would just open two connections: one for JDBC work and one
> for file transfer. Keep them separate.
That may be the simpler approach.
Dan.
Re: advice for client/server application
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Yes, I see what you're saying, you're rewriting JDBC extended with file
transfer.
You could take the Derby source and extend it so that you can do file
transfer functionality, but then you're stuck with your own fork of
JDBC. And I'm sorry to say that the Derby community would likely not
accept your changes back (someone correct me if I'm wrong) because there
would be non-standard extensions to JDBC, and we strive to be
standards-compliant as a community.
Personally, I would just open two connections: one for JDBC work and one
for file transfer. Keep them separate.
David
Ryan P Bobko wrote:
> Thanks for the advice, but I'm not sure if that will help. I now realize what
> makes me uneasy about my architecture is that I feel like I'm rewriting JDBC
> little by little just so I can have my file-moving piece on top of it. JDBC
> works great for my purposes, so my preference would be to remove my custom
> protocol whenever possible. Is it feasible to extend the JDBC API? Or will
> that just be more trouble than it's worth?
>
> I realize I just changed my question. Any more advice?
>
> ry
>
>
> On Wednesday 22 March 2006 07:23 pm, David W. Van Couvering wrote:
>
>>A common way client applications working with large result sets have
>>handled the "too much memory" problem that I've seen is to send the
>>results over in chunks. Instead of sending all 120,000 records in one
>>response, just send 100 or 1,000. The client processes those 1,000
>>records, throws them away, and get the next 1,000.
>>
>>Would that work for you?
>>
>>David
>>
>>Ryan P Bobko wrote:
>>
>>>Hi List,
>>>First of all, I can't say enough how impressed I've been with Derby.
>>>Every time I've thought this embedded wouldn't be able to do something I
>>>expect from a "full-blown" database (nested selects, correlated
>>>subqueries, stored procedures, you name it), it's suprised me. I love it.
>>>
>>>This isn't strictly a Derby question, but I'm hoping for some advice or
>>>suggestions with how to procede. I've been working on an application that
>>>is a sort of half-database, half-FTP client/server setup. The protocol
>>>I've implemented between the client and server lets the app do things
>>>like run queries, but also move files around based on those results. Or
>>>insert rows into the database based on where files have moved to. Files
>>>can be moved from the server to client and vice versa.
>>>
>>>Things have been working just fine, except that when I run queries, the
>>>server process does all the work and returns the results as a vector of
>>>string arrays. It's never sat well with me--and as you can imagine--now
>>>that the dataset is getting pretty big (120.000-4KB rows returned for
>>>some queries), I'm using too much memory.
>>>
>>>What I'd like to do is get my jdbc connection object onto the client so I
>>>don't have to "package" everything up when returning resultsets. The
>>>question is how? My first idea was to just use derby's network server
>>>and write the file protocol separately, but I'd prefer to stick with just
>>>one socket if I can.
>>>
>>>Advice? Thanks for your time.
>>>ry
>
>
Re: advice for client/server application
Posted by Ryan P Bobko <ry...@ostrich-emulators.com>.
I've often wondered about why one would store files in the db, too. The main
reason I've thought of is the you don't run into inconsistencies between your
database information and your file repository. You get the benefit of
transactions and rollbacks, for example. Otherwise, you need to keep the db
and files in sync via some other means.
That said, I took the advice to be using blobs to get the files to the server
and stick them in a temporary (in a logical sense) table. Then one could
execute a stored procedure to take the file out of the db and stick it in the
file repository. Same goes for retrieving files.
Thanks again,
ry
On Friday 24 March 2006 03:48 pm, Michael Segel wrote:
> On Friday 24 March 2006 12:44 am, Veaceslav Chicu wrote:
> > if the problem is to move files, you can do this with blobs?
> >
> > you can have a stored procedures that will save, read file on the server
> >
> >
> > best regards,
> > Slavic
>
> I was wondering about that.
> Derby does support IO Streams so you could do that.
> But this sort of begs the question... why store the file in the database?
>
> If you look at content management solutions from companies like IBM their
> solutions are based on storing the file's meta data within the database.
>
> If you take that design approach, you'll find that you can separate your
> TFTP/FTP issues away from the JDBC layer.
>
> But this doesn't answer the question of why you would want to store the
> file within the database in the first place?
>
> Going back up the thread... the original poster writes:
> "Things have been working just fine, except that when I run queries, the
>
> >>>>>server process does all the work and returns the results as a vector
> >>>>> of string arrays. It's never sat well with me--and as you can
> >>>>> imagine--now that the dataset is getting pretty big (120.000-4KB rows
> >>>>> returned for some queries), I'm using too much memory.
> >>>>>
> >>>>>What I'd like to do is get my jdbc connection object onto the client
> >>>>> so I don't have to "package" everything up when returning resultsets.
> >>>>> The question is how? My first idea was to just use derby's network
> >>>>> server and write the file protocol separately, but I'd prefer to
> >>>>> stick with just one socket if I can.
>
> "
>
> This may not make sense... 4KB objects to contain the meta data information
> regarding files? (Author says app is 1/2 ftp , 1/2 database app.)
>
> IMHO, there is a design issue that needs to be worked out prior to trying
> to code up the application. It could be that you're trying to fit a square
> peg in a round hole....
>
> > Ryan P Bobko wrote:
> > > Thanks for the advice, but I'm not sure if that will help. I now
> > > realize what makes me uneasy about my architecture is that I feel like
> > > I'm rewriting JDBC little by little just so I can have my file-moving
> > > piece on top of it. JDBC works great for my purposes, so my preference
> > > would be to remove my custom protocol whenever possible. Is it feasible
> > > to extend the JDBC API? Or will that just be more trouble than it's
> > > worth?
> > >
> > > I realize I just changed my question. Any more advice?
> > >
> > > ry
> > >
> > > On Wednesday 22 March 2006 07:23 pm, David W. Van Couvering wrote:
> > >> A common way client applications working with large result sets have
> > >> handled the "too much memory" problem that I've seen is to send the
> > >> results over in chunks. Instead of sending all 120,000 records in one
> > >> response, just send 100 or 1,000. The client processes those 1,000
> > >> records, throws them away, and get the next 1,000.
> > >>
> > >> Would that work for you?
> > >>
> > >> David
> > >>
> > >> Ryan P Bobko wrote:
> > >>> Hi List,
> > >>> First of all, I can't say enough how impressed I've been with Derby.
> > >>> Every time I've thought this embedded wouldn't be able to do
> > >>> something I expect from a "full-blown" database (nested selects,
> > >>> correlated subqueries, stored procedures, you name it), it's suprised
> > >>> me. I love it.
> > >>>
> > >>> This isn't strictly a Derby question, but I'm hoping for some advice
> > >>> or suggestions with how to procede. I've been working on an
> > >>> application that is a sort of half-database, half-FTP client/server
> > >>> setup. The protocol I've implemented between the client and server
> > >>> lets the app do things like run queries, but also move files around
> > >>> based on those results. Or insert rows into the database based on
> > >>> where files have moved to. Files can be moved from the server to
> > >>> client and vice versa.
> > >>>
> > >>> Things have been working just fine, except that when I run queries,
> > >>> the server process does all the work and returns the results as a
> > >>> vector of string arrays. It's never sat well with me--and as you can
> > >>> imagine--now that the dataset is getting pretty big (120.000-4KB rows
> > >>> returned for some queries), I'm using too much memory.
> > >>>
> > >>> What I'd like to do is get my jdbc connection object onto the client
> > >>> so I don't have to "package" everything up when returning resultsets.
> > >>> The question is how? My first idea was to just use derby's network
> > >>> server and write the file protocol separately, but I'd prefer to
> > >>> stick with just one socket if I can.
> > >>>
> > >>> Advice? Thanks for your time.
> > >>> ry
--
Go not to the elves for counsel, for they will say both yes and no.
-- J.R.R. Tolkien
Re: advice for client/server application
Posted by Veaceslav Chicu <cs...@infologic.fr>.
>
> I was wondering about that.
> Derby does support IO Streams so you could do that.
> But this sort of begs the question... why store the file in the database?
>
> If you look at content management solutions from companies like IBM their
> solutions are based on storing the file's meta data within the database.
>
> If you take that design approach, you'll find that you can separate your
> TFTP/FTP issues away from the JDBC layer.
>
in DB2 they have a special infrastructure to keep in sync files with
database reference, it's not so simple
the simplest way is to have a table with blobs
but if you want simple to upload download files, maybe you can write 2
stored procedures
{call upload_file(VARCHAR(254) filename, byte[] file)}
{call download_file(VARCHAR(254)filename)}
in this procedure you just store the file or read file from file system,
maybe you need a procedure to look at the list of files in directory
if you have a webapp and files a images for example, it's better to keep
just references in the database, and implement some methods to keep in
sync with database
best regards,
Slavic
Re: advice for client/server application
Posted by Michael Segel <de...@segel.com>.
On Friday 24 March 2006 12:44 am, Veaceslav Chicu wrote:
> if the problem is to move files, you can do this with blobs?
>
> you can have a stored procedures that will save, read file on the server
>
>
> best regards,
> Slavic
>
I was wondering about that.
Derby does support IO Streams so you could do that.
But this sort of begs the question... why store the file in the database?
If you look at content management solutions from companies like IBM their
solutions are based on storing the file's meta data within the database.
If you take that design approach, you'll find that you can separate your
TFTP/FTP issues away from the JDBC layer.
But this doesn't answer the question of why you would want to store the file
within the database in the first place?
Going back up the thread... the original poster writes:
"Things have been working just fine, except that when I run queries, the
>>>>>server process does all the work and returns the results as a vector of
>>>>>string arrays. It's never sat well with me--and as you can imagine--now
>>>>>that the dataset is getting pretty big (120.000-4KB rows returned for
>>>>>some queries), I'm using too much memory.
>>>>>
>>>>>What I'd like to do is get my jdbc connection object onto the client so
>>>>>I don't have to "package" everything up when returning resultsets. The
>>>>>question is how? My first idea was to just use derby's network server
>>>>>and write the file protocol separately, but I'd prefer to stick with
>>>>>just one socket if I can.
"
This may not make sense... 4KB objects to contain the meta data information
regarding files? (Author says app is 1/2 ftp , 1/2 database app.)
IMHO, there is a design issue that needs to be worked out prior to trying to
code up the application. It could be that you're trying to fit a square peg
in a round hole....
> Ryan P Bobko wrote:
> > Thanks for the advice, but I'm not sure if that will help. I now realize
> > what makes me uneasy about my architecture is that I feel like I'm
> > rewriting JDBC little by little just so I can have my file-moving piece
> > on top of it. JDBC works great for my purposes, so my preference would be
> > to remove my custom protocol whenever possible. Is it feasible to extend
> > the JDBC API? Or will that just be more trouble than it's worth?
> >
> > I realize I just changed my question. Any more advice?
> >
> > ry
> >
> > On Wednesday 22 March 2006 07:23 pm, David W. Van Couvering wrote:
> >> A common way client applications working with large result sets have
> >> handled the "too much memory" problem that I've seen is to send the
> >> results over in chunks. Instead of sending all 120,000 records in one
> >> response, just send 100 or 1,000. The client processes those 1,000
> >> records, throws them away, and get the next 1,000.
> >>
> >> Would that work for you?
> >>
> >> David
> >>
> >> Ryan P Bobko wrote:
> >>> Hi List,
> >>> First of all, I can't say enough how impressed I've been with Derby.
> >>> Every time I've thought this embedded wouldn't be able to do something
> >>> I expect from a "full-blown" database (nested selects, correlated
> >>> subqueries, stored procedures, you name it), it's suprised me. I love
> >>> it.
> >>>
> >>> This isn't strictly a Derby question, but I'm hoping for some advice or
> >>> suggestions with how to procede. I've been working on an application
> >>> that is a sort of half-database, half-FTP client/server setup. The
> >>> protocol I've implemented between the client and server lets the app do
> >>> things like run queries, but also move files around based on those
> >>> results. Or insert rows into the database based on where files have
> >>> moved to. Files can be moved from the server to client and vice versa.
> >>>
> >>> Things have been working just fine, except that when I run queries, the
> >>> server process does all the work and returns the results as a vector of
> >>> string arrays. It's never sat well with me--and as you can imagine--now
> >>> that the dataset is getting pretty big (120.000-4KB rows returned for
> >>> some queries), I'm using too much memory.
> >>>
> >>> What I'd like to do is get my jdbc connection object onto the client so
> >>> I don't have to "package" everything up when returning resultsets. The
> >>> question is how? My first idea was to just use derby's network server
> >>> and write the file protocol separately, but I'd prefer to stick with
> >>> just one socket if I can.
> >>>
> >>> Advice? Thanks for your time.
> >>> ry
--
--
Michael Segel
Principal
Michael Segel Consulting Corp.
derby@segel.com
(312) 952-8175 [mobile]
Re: advice for client/server application
Posted by Veaceslav Chicu <cs...@infologic.fr>.
if the problem is to move files, you can do this with blobs?
you can have a stored procedures that will save, read file on the server
best regards,
Slavic
Ryan P Bobko wrote:
> Thanks for the advice, but I'm not sure if that will help. I now realize what
> makes me uneasy about my architecture is that I feel like I'm rewriting JDBC
> little by little just so I can have my file-moving piece on top of it. JDBC
> works great for my purposes, so my preference would be to remove my custom
> protocol whenever possible. Is it feasible to extend the JDBC API? Or will
> that just be more trouble than it's worth?
>
> I realize I just changed my question. Any more advice?
>
> ry
>
>
> On Wednesday 22 March 2006 07:23 pm, David W. Van Couvering wrote:
>> A common way client applications working with large result sets have
>> handled the "too much memory" problem that I've seen is to send the
>> results over in chunks. Instead of sending all 120,000 records in one
>> response, just send 100 or 1,000. The client processes those 1,000
>> records, throws them away, and get the next 1,000.
>>
>> Would that work for you?
>>
>> David
>>
>> Ryan P Bobko wrote:
>>> Hi List,
>>> First of all, I can't say enough how impressed I've been with Derby.
>>> Every time I've thought this embedded wouldn't be able to do something I
>>> expect from a "full-blown" database (nested selects, correlated
>>> subqueries, stored procedures, you name it), it's suprised me. I love it.
>>>
>>> This isn't strictly a Derby question, but I'm hoping for some advice or
>>> suggestions with how to procede. I've been working on an application that
>>> is a sort of half-database, half-FTP client/server setup. The protocol
>>> I've implemented between the client and server lets the app do things
>>> like run queries, but also move files around based on those results. Or
>>> insert rows into the database based on where files have moved to. Files
>>> can be moved from the server to client and vice versa.
>>>
>>> Things have been working just fine, except that when I run queries, the
>>> server process does all the work and returns the results as a vector of
>>> string arrays. It's never sat well with me--and as you can imagine--now
>>> that the dataset is getting pretty big (120.000-4KB rows returned for
>>> some queries), I'm using too much memory.
>>>
>>> What I'd like to do is get my jdbc connection object onto the client so I
>>> don't have to "package" everything up when returning resultsets. The
>>> question is how? My first idea was to just use derby's network server
>>> and write the file protocol separately, but I'd prefer to stick with just
>>> one socket if I can.
>>>
>>> Advice? Thanks for your time.
>>> ry
>
Re: advice for client/server application
Posted by Ryan P Bobko <ry...@ostrich-emulators.com>.
Thanks for the advice, but I'm not sure if that will help. I now realize what
makes me uneasy about my architecture is that I feel like I'm rewriting JDBC
little by little just so I can have my file-moving piece on top of it. JDBC
works great for my purposes, so my preference would be to remove my custom
protocol whenever possible. Is it feasible to extend the JDBC API? Or will
that just be more trouble than it's worth?
I realize I just changed my question. Any more advice?
ry
On Wednesday 22 March 2006 07:23 pm, David W. Van Couvering wrote:
> A common way client applications working with large result sets have
> handled the "too much memory" problem that I've seen is to send the
> results over in chunks. Instead of sending all 120,000 records in one
> response, just send 100 or 1,000. The client processes those 1,000
> records, throws them away, and get the next 1,000.
>
> Would that work for you?
>
> David
>
> Ryan P Bobko wrote:
> > Hi List,
> > First of all, I can't say enough how impressed I've been with Derby.
> > Every time I've thought this embedded wouldn't be able to do something I
> > expect from a "full-blown" database (nested selects, correlated
> > subqueries, stored procedures, you name it), it's suprised me. I love it.
> >
> > This isn't strictly a Derby question, but I'm hoping for some advice or
> > suggestions with how to procede. I've been working on an application that
> > is a sort of half-database, half-FTP client/server setup. The protocol
> > I've implemented between the client and server lets the app do things
> > like run queries, but also move files around based on those results. Or
> > insert rows into the database based on where files have moved to. Files
> > can be moved from the server to client and vice versa.
> >
> > Things have been working just fine, except that when I run queries, the
> > server process does all the work and returns the results as a vector of
> > string arrays. It's never sat well with me--and as you can imagine--now
> > that the dataset is getting pretty big (120.000-4KB rows returned for
> > some queries), I'm using too much memory.
> >
> > What I'd like to do is get my jdbc connection object onto the client so I
> > don't have to "package" everything up when returning resultsets. The
> > question is how? My first idea was to just use derby's network server
> > and write the file protocol separately, but I'd prefer to stick with just
> > one socket if I can.
> >
> > Advice? Thanks for your time.
> > ry
--
"Son I am able," she said, "though you scare me."
"Watch," said I, "Beloved." I said, "Watch me scare you though."
Said she: "Able am I son."
-- They Might Be Giants
Re: advice for client/server application
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
A common way client applications working with large result sets have
handled the "too much memory" problem that I've seen is to send the
results over in chunks. Instead of sending all 120,000 records in one
response, just send 100 or 1,000. The client processes those 1,000
records, throws them away, and get the next 1,000.
Would that work for you?
David
Ryan P Bobko wrote:
> Hi List,
> First of all, I can't say enough how impressed I've been with Derby. Every
> time I've thought this embedded wouldn't be able to do something I expect
> from a "full-blown" database (nested selects, correlated subqueries, stored
> procedures, you name it), it's suprised me. I love it.
>
> This isn't strictly a Derby question, but I'm hoping for some advice or
> suggestions with how to procede. I've been working on an application that is
> a sort of half-database, half-FTP client/server setup. The protocol I've
> implemented between the client and server lets the app do things like run
> queries, but also move files around based on those results. Or insert rows
> into the database based on where files have moved to. Files can be moved from
> the server to client and vice versa.
>
> Things have been working just fine, except that when I run queries, the server
> process does all the work and returns the results as a vector of string
> arrays. It's never sat well with me--and as you can imagine--now that the
> dataset is getting pretty big (120.000-4KB rows returned for some queries),
> I'm using too much memory.
>
> What I'd like to do is get my jdbc connection object onto the client so I
> don't have to "package" everything up when returning resultsets. The question
> is how? My first idea was to just use derby's network server and write the
> file protocol separately, but I'd prefer to stick with just one socket if I
> can.
>
> Advice? Thanks for your time.
> ry
>