You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-user@db.apache.org by "Fantry, John" <JF...@analogic.com> on 2006/07/26 16:37:12 UTC
How to monitor progress during data import
I am using Derby to import a large text file into a table. I would like
to be able to display a progress dialog to the user while the import is
processing. In order to provide a useful progress bar I need to be able
to calculate the % complete of the import procedure. Does anyone know
if this can be done? The table is locked during the import so I can't
execute any SELECT COUNT(*) statements in a separate thread. Perhaps
Derby outputs some statistics to a log file? I did stumble across the
RunTimeStatistics attribute in the Derby docs, but this attribute is
only meaningful after the import is complete. I need insight into the
import procedure while it's executing. Any ideas?
Thanks,
-John
****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.
Thank you.
RE: How to monitor progress during data import
Posted by "Fantry, John" <JF...@analogic.com>.
I am importing the data using the built in system procedure
SYSCS_UTIL.SYSCS_IMPORT_TABLE. This procedure takes the name of the
file to import as an argument. Derby opens the file using whatever
mechanism it wants. I don't have an opportunity to wrap the stream
myself.
-----Original Message-----
From: Mike Matrigali [mailto:mikem_app@sbcglobal.net]
Sent: Wednesday, July 26, 2006 11:27 AM
To: Derby Discussion
Subject: Re: How to monitor progress during data import
Andreas Korneliussen wrote:
> Fantry, John wrote:
>
>> I am using Derby to import a large text file into a table. I would
>> like to be able to display a progress dialog to the user while the
>> import is processing. In order to provide a useful progress bar I
>> need to be able to calculate the % complete of the import procedure.
>> Does anyone know if this can be done? The table is locked during the
>> import so I can't execute any SELECT COUNT(*) statements in a
separate
>> thread. Perhaps Derby outputs some statistics to a log file? I did
>> stumble across the RunTimeStatistics attribute in the Derby docs, but
>> this attribute is only meaningful after the import is complete. I
>> need insight into the import procedure while it's executing. Any
ideas?
>>
>
>
> Have you tried running the SELECT COUNT (*) statement using
transaction
> isolation level "read-uncommitted " ?
Do note that derby must scan all the rows in the table to execute
count(*), so checking on the progress using count(*) is a significant
overhead depending on the size of the table.
For your application it may be possible to layer a stream on top of the
file which could tell your app how much has been read while otherwise
just passing the bytes from the file to derby.
>
> Regards
> Andreas
>
>
>
****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.
Thank you.
Re: How to monitor progress during data import
Posted by Mike Matrigali <mi...@sbcglobal.net>.
Andreas Korneliussen wrote:
> Fantry, John wrote:
>
>> I am using Derby to import a large text file into a table. I would
>> like to be able to display a progress dialog to the user while the
>> import is processing. In order to provide a useful progress bar I
>> need to be able to calculate the % complete of the import procedure.
>> Does anyone know if this can be done? The table is locked during the
>> import so I can't execute any SELECT COUNT(*) statements in a separate
>> thread. Perhaps Derby outputs some statistics to a log file? I did
>> stumble across the RunTimeStatistics attribute in the Derby docs, but
>> this attribute is only meaningful after the import is complete. I
>> need insight into the import procedure while it's executing. Any ideas?
>>
>
>
> Have you tried running the SELECT COUNT (*) statement using transaction
> isolation level "read-uncommitted " ?
Do note that derby must scan all the rows in the table to execute
count(*), so checking on the progress using count(*) is a significant
overhead depending on the size of the table.
For your application it may be possible to layer a stream on top of the
file which could tell your app how much has been read while otherwise
just passing the bytes from the file to derby.
>
> Regards
> Andreas
>
>
>
Re: How to monitor progress during data import
Posted by Andreas Korneliussen <An...@Sun.COM>.
Fantry, John wrote:
> I am using Derby to import a large text file into a table. I would like
> to be able to display a progress dialog to the user while the import is
> processing. In order to provide a useful progress bar I need to be able
> to calculate the % complete of the import procedure. Does anyone know
> if this can be done? The table is locked during the import so I can't
> execute any SELECT COUNT(*) statements in a separate thread. Perhaps
> Derby outputs some statistics to a log file? I did stumble across the
> RunTimeStatistics attribute in the Derby docs, but this attribute is
> only meaningful after the import is complete. I need insight into the
> import procedure while it's executing. Any ideas?
>
Have you tried running the SELECT COUNT (*) statement using transaction
isolation level "read-uncommitted " ?
Regards
Andreas