You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oodt.apache.org by Thomas Bennett <lm...@gmail.com> on 2012/07/16 16:25:45 UTC
How can I update a filemanager's catalogue?
Hi,
I've recently created a Lucene file manager catalog. I cataloged about 2000
files "in place", but have encountered a problem.
>From the fmbrowser some of the files are listed as being:
Product Transfer Status Percent Complete Received Time References
Metadata
1287587805.h5 RECEIVED 100% UNKNOWN View References
View Metadata
1287353769.h5 RECEIVED *0%* UNKNOWN View
References View Metadata
1287355996.h5 RECEIVED *0%* UNKNOWN View
References View Metadata
1287425397.h5 RECEIVED 100% UNKNOWN View References
View Metadata
The problem lies with the two products that show % complete being 0%.
When I use fmprod to download the product:
wget http://..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e<http://kat-archive.kat.ac.za:8983/fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e>
I get:
HTTP request sent, awaiting response... 500 Internal Server Error
2012-07-14 20:20:38 ERROR 500: Internal Server Error.
So I'd like to "edit" the lucene catalog to update this piece of metadata
to reflect that the % complete is 100. Are there any tools available to do
this?
Also at some point I must figure out why its happening :)
Cheers,
Tom
Re: How can I update a filemanager's catalogue?
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Tom,
On Jul 16, 2012, at 9:25 AM, Thomas Bennett wrote:
> Hi,
>
> I've recently created a Lucene file manager catalog. I cataloged about 2000 files "in place", but have encountered a problem.
>
> From the fmbrowser some of the files are listed as being:
>
> Product Transfer Status Percent Complete Received Time References Metadata
> 1287587805.h5 RECEIVED 100% UNKNOWN View References View Metadata
> 1287353769.h5 RECEIVED 0% UNKNOWN View References View Metadata
> 1287355996.h5 RECEIVED 0% UNKNOWN View References View Metadata
> 1287425397.h5 RECEIVED 100% UNKNOWN View References View Metadata
>
> The problem lies with the two products that show % complete being 0%.
This is because the data transfer status tracker, and the combination of it, and the data transferer
selected -- InPlaceDataTransferer doesn't notify the transfer status tracker and also doesn't update the
transfer to be 100% done, IIRC.
>
> When I use fmprod to download the product:
>
> wget http://..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e
>
> I get:
> HTTP request sent, awaiting response... 500 Internal Server Error
> 2012-07-14 20:20:38 ERROR 500: Internal Server Error.
Yep. This is also likely due to the fact that the content size (aka FileSize) might not be set as well
b/c of the InPlaceDataTransfer and InPlaceVersioner.
If you remember, Tom, can you file a bug here so I'll remember (or someone else will) to come by
and fix it for 0.5?
>
> So I'd like to "edit" the lucene catalog to update this piece of metadata to reflect that the % complete is 100. Are there any tools available to do this?
Luke is the tool to use to directly edit the Lucene catalog. Unfortunately b/c the FM depends on such an old version
of Lucene, we need a way old (0.6.2 I think or 0.7.0) version of lukeall.jar to support this. In the meanwhile,
that's a stop gap, but let's try and upgrade OODT to use a newer version of Lucene for 0.5.
>
> Also at some point I must figure out why its happening :)
Yep we'll fix it! :)
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: How can I update a filemanager's catalogue?
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Tom,
On Jul 16, 2012, at 9:25 AM, Thomas Bennett wrote:
> Hi,
>
> I've recently created a Lucene file manager catalog. I cataloged about 2000 files "in place", but have encountered a problem.
>
> From the fmbrowser some of the files are listed as being:
>
> Product Transfer Status Percent Complete Received Time References Metadata
> 1287587805.h5 RECEIVED 100% UNKNOWN View References View Metadata
> 1287353769.h5 RECEIVED 0% UNKNOWN View References View Metadata
> 1287355996.h5 RECEIVED 0% UNKNOWN View References View Metadata
> 1287425397.h5 RECEIVED 100% UNKNOWN View References View Metadata
>
> The problem lies with the two products that show % complete being 0%.
This is because the data transfer status tracker, and the combination of it, and the data transferer
selected -- InPlaceDataTransferer doesn't notify the transfer status tracker and also doesn't update the
transfer to be 100% done, IIRC.
>
> When I use fmprod to download the product:
>
> wget http://..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e
>
> I get:
> HTTP request sent, awaiting response... 500 Internal Server Error
> 2012-07-14 20:20:38 ERROR 500: Internal Server Error.
Yep. This is also likely due to the fact that the content size (aka FileSize) might not be set as well
b/c of the InPlaceDataTransfer and InPlaceVersioner.
If you remember, Tom, can you file a bug here so I'll remember (or someone else will) to come by
and fix it for 0.5?
>
> So I'd like to "edit" the lucene catalog to update this piece of metadata to reflect that the % complete is 100. Are there any tools available to do this?
Luke is the tool to use to directly edit the Lucene catalog. Unfortunately b/c the FM depends on such an old version
of Lucene, we need a way old (0.6.2 I think or 0.7.0) version of lukeall.jar to support this. In the meanwhile,
that's a stop gap, but let's try and upgrade OODT to use a newer version of Lucene for 0.5.
>
> Also at some point I must figure out why its happening :)
Yep we'll fix it! :)
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: How can I update a filemanager's catalogue?
Posted by Sheryl John <sh...@gmail.com>.
Hi Tom,
I'm not sure if modifying the metadata will help you update the 'percent
complete' status of the file.
Since this is calculated by the filemgr (FileTransferStatus), maybe there
was something wrong with the file size or uploading.
On Mon, Jul 16, 2012 at 7:25 AM, Thomas Bennett <lm...@gmail.com> wrote:
> Hi,
>
> I've recently created a Lucene file manager catalog. I cataloged about
> 2000 files "in place", but have encountered a problem.
>
> From the fmbrowser some of the files are listed as being:
>
> Product Transfer Status Percent Complete Received Time
> References Metadata
> 1287587805.h5 RECEIVED 100% UNKNOWN View References
> View Metadata
> 1287353769.h5 RECEIVED *0%* UNKNOWN View
> References View Metadata
> 1287355996.h5 RECEIVED *0%* UNKNOWN View
> References View Metadata
> 1287425397.h5 RECEIVED 100% UNKNOWN View References
> View Metadata
>
> The problem lies with the two products that show % complete being 0%.
>
> When I use fmprod to download the product:
>
> wget
> http://..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e<http://kat-archive.kat.ac.za:8983/fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e>
>
> I get:
> HTTP request sent, awaiting response... 500 Internal Server Error
> 2012-07-14 20:20:38 ERROR 500: Internal Server Error.
>
> So I'd like to "edit" the lucene catalog to update this piece of metadata
> to reflect that the % complete is 100. Are there any tools available to do
> this?
>
> Also at some point I must figure out why its happening :)
>
> Cheers,
> Tom
>
--
-Sheryl
Re: How can I update a filemanager's catalogue?
Posted by Cameron Goodale <si...@gmail.com>.
Sheryl,
Thanks for the warning. Paul and I stumbled onto the Luke scripting feature
but didn't use them.
Lucene catalog cleansing is something we will be addressing for the snow
project. If anything useful comes out of it, we'll be sure to share.
Cameron
On Jul 16, 2012 9:40 AM, "Sheryl John" <sh...@gmail.com> wrote:
> Hi Cameron,
>
> Were you able to edit the filemgr lucene catalog?
> I think if you update an index with Luke it would write out a version that
> can't be read by an older Solr/Lucene version.
> I remember Ricky trying to edit the filemgr catalog with Luke and faced
> the above issue.
>
> On Mon, Jul 16, 2012 at 8:09 AM, Cameron Goodale <si...@gmail.com>wrote:
>
>> Tom,
>>
>> You could use lucene's Luke tool to query and edit the catalog. Paul and
>> I discovered a scripting plugin within Luke that can execute javascript to
>> manipulate the catalog.
>>
>> I hope that is a good lead for you.
>>
>> Cameron
>> On Jul 16, 2012 8:26 AM, "Thomas Bennett" <lm...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I've recently created a Lucene file manager catalog. I cataloged about
>>> 2000 files "in place", but have encountered a problem.
>>>
>>> From the fmbrowser some of the files are listed as being:
>>>
>>> Product Transfer Status Percent Complete Received Time
>>> References Metadata
>>> 1287587805.h5 RECEIVED 100% UNKNOWN View References
>>> View Metadata
>>> 1287353769.h5 RECEIVED *0%* UNKNOWN View
>>> References View Metadata
>>> 1287355996.h5 RECEIVED *0%* UNKNOWN View
>>> References View Metadata
>>> 1287425397.h5 RECEIVED 100% UNKNOWN View References
>>> View Metadata
>>>
>>> The problem lies with the two products that show % complete being 0%.
>>>
>>> When I use fmprod to download the product:
>>>
>>> wget
>>> http://..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e<http://kat-archive.kat.ac.za:8983/fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e>
>>>
>>> I get:
>>> HTTP request sent, awaiting response... 500 Internal Server Error
>>> 2012-07-14 20:20:38 ERROR 500: Internal Server Error.
>>>
>>> So I'd like to "edit" the lucene catalog to update this piece of
>>> metadata to reflect that the % complete is 100. Are there any tools
>>> available to do this?
>>>
>>> Also at some point I must figure out why its happening :)
>>>
>>> Cheers,
>>> Tom
>>>
>>
>
>
> --
> -Sheryl
>
Re: How can I update a filemanager's catalogue?
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Ricky,
On Jul 16, 2012, at 1:43 PM, Nguyen, Ricky wrote:
> Awhile back, I wrote a tiny Java main program to edit a Lucene catalog (attempting to clean our catalog), but discovered that once you write out v3.x, you can't read it back in with OODT.
Yep that's b/c Apache OODT uses 2.0 as you state below and the *modern* version of Luke uses 3.x. You can find older versions (e.g., 0.6.2 or 0.7) lukeall.jars that are compatible with older formats, but YMMV.
Thanks!
Cheers,
Chris
P.S. Yes this sucks and we need to upgrade to the newest version of Lucene :) If no one beats me to this in 0.5 I'll do it myself :)
>
> On Jul 16, 2012, at 11:32 AM, Thomas Bennett wrote:
>
>> Hey Sheryl and Cameron,
>>
>> I have downloaded Luke with the intention of trying it out. Thanks for the warning Sheryl. I'll give it a try unless Ricky advises otherwise.
>>
>> Yeah, the files are all okay i.e. have size and content, so I suspect a bug of some sort.
>>
>> I'm planing to take a look to see how file transfer status is updated. Since I'm using the ingest with the InPlaceDataTrasfer (I think that's the option for the crawler) it may have introduces an interesting aspect since the file is not actually transferred :)
>>
>> Cheers,
>> Tom
>>
>> On 16 July 2012 17:40, Sheryl John <sh...@gmail.com> wrote:
>> Hi Cameron,
>>
>> Were you able to edit the filemgr lucene catalog?
>> I think if you update an index with Luke it would write out a version that can't be read by an older Solr/Lucene version.
>> I remember Ricky trying to edit the filemgr catalog with Luke and faced the above issue.
>>
>>
>> On Mon, Jul 16, 2012 at 8:09 AM, Cameron Goodale <si...@gmail.com> wrote:
>> Tom,
>>
>> You could use lucene's Luke tool to query and edit the catalog. Paul and I discovered a scripting plugin within Luke that can execute javascript to manipulate the catalog.
>>
>> I hope that is a good lead for you.
>>
>> Cameron
>>
>> On Jul 16, 2012 8:26 AM, "Thomas Bennett" <lm...@gmail.com> wrote:
>> Hi,
>>
>> I've recently created a Lucene file manager catalog. I cataloged about 2000 files "in place", but have encountered a problem.
>>
>> From the fmbrowser some of the files are listed as being:
>>
>> Product
>> Transfer Status Percent Complete
>> Received Time References
>> Metadata
>> 1287587805.h5
>> RECEIVED 100%
>> UNKNOWN View References
>> View Metadata
>> 1287353769.h5
>> RECEIVED 0%
>> UNKNOWN View References
>> View Metadata
>> 1287355996.h5
>> RECEIVED 0%
>> UNKNOWN View References
>> View Metadata
>> 1287425397.h5
>> RECEIVED 100%
>> UNKNOWN View References
>> View Metadata
>>
>> The problem lies with the two products that show % complete being 0%.
>>
>> When I use fmprod to download the product:
>>
>> wget http://..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e
>>
>> I get:
>> HTTP request sent, awaiting response... 500 Internal Server Error
>> 2012-07-14 20:20:38 ERROR 500: Internal Server Error.
>>
>> So I'd like to "edit" the lucene catalog to update this piece of metadata to reflect that the % complete is 100. Are there any tools available to do this?
>>
>> Also at some point I must figure out why its happening :)
>>
>> Cheers,
>> Tom
>>
>>
>>
>> --
>> -Sheryl
>>
>
>
> ---------------------------------------------------------------------
> CONFIDENTIALITY NOTICE: This e-mail message, including any attachments,
> is for the sole use of the intended recipient(s) and may contain confidential
> or legally privileged information. Any unauthorized review, use, disclosure
> or distribution is prohibited. If you are not the intended recipient, please
> contact the sender by reply e-mail and destroy all copies of this original message.
>
> ---------------------------------------------------------------------
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: How can I update a filemanager's catalogue?
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Ricky,
On Jul 16, 2012, at 1:43 PM, Nguyen, Ricky wrote:
> Awhile back, I wrote a tiny Java main program to edit a Lucene catalog (attempting to clean our catalog), but discovered that once you write out v3.x, you can't read it back in with OODT.
Yep that's b/c Apache OODT uses 2.0 as you state below and the *modern* version of Luke uses 3.x. You can find older versions (e.g., 0.6.2 or 0.7) lukeall.jars that are compatible with older formats, but YMMV.
Thanks!
Cheers,
Chris
P.S. Yes this sucks and we need to upgrade to the newest version of Lucene :) If no one beats me to this in 0.5 I'll do it myself :)
>
> On Jul 16, 2012, at 11:32 AM, Thomas Bennett wrote:
>
>> Hey Sheryl and Cameron,
>>
>> I have downloaded Luke with the intention of trying it out. Thanks for the warning Sheryl. I'll give it a try unless Ricky advises otherwise.
>>
>> Yeah, the files are all okay i.e. have size and content, so I suspect a bug of some sort.
>>
>> I'm planing to take a look to see how file transfer status is updated. Since I'm using the ingest with the InPlaceDataTrasfer (I think that's the option for the crawler) it may have introduces an interesting aspect since the file is not actually transferred :)
>>
>> Cheers,
>> Tom
>>
>> On 16 July 2012 17:40, Sheryl John <sh...@gmail.com> wrote:
>> Hi Cameron,
>>
>> Were you able to edit the filemgr lucene catalog?
>> I think if you update an index with Luke it would write out a version that can't be read by an older Solr/Lucene version.
>> I remember Ricky trying to edit the filemgr catalog with Luke and faced the above issue.
>>
>>
>> On Mon, Jul 16, 2012 at 8:09 AM, Cameron Goodale <si...@gmail.com> wrote:
>> Tom,
>>
>> You could use lucene's Luke tool to query and edit the catalog. Paul and I discovered a scripting plugin within Luke that can execute javascript to manipulate the catalog.
>>
>> I hope that is a good lead for you.
>>
>> Cameron
>>
>> On Jul 16, 2012 8:26 AM, "Thomas Bennett" <lm...@gmail.com> wrote:
>> Hi,
>>
>> I've recently created a Lucene file manager catalog. I cataloged about 2000 files "in place", but have encountered a problem.
>>
>> From the fmbrowser some of the files are listed as being:
>>
>> Product
>> Transfer Status Percent Complete
>> Received Time References
>> Metadata
>> 1287587805.h5
>> RECEIVED 100%
>> UNKNOWN View References
>> View Metadata
>> 1287353769.h5
>> RECEIVED 0%
>> UNKNOWN View References
>> View Metadata
>> 1287355996.h5
>> RECEIVED 0%
>> UNKNOWN View References
>> View Metadata
>> 1287425397.h5
>> RECEIVED 100%
>> UNKNOWN View References
>> View Metadata
>>
>> The problem lies with the two products that show % complete being 0%.
>>
>> When I use fmprod to download the product:
>>
>> wget http://..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e
>>
>> I get:
>> HTTP request sent, awaiting response... 500 Internal Server Error
>> 2012-07-14 20:20:38 ERROR 500: Internal Server Error.
>>
>> So I'd like to "edit" the lucene catalog to update this piece of metadata to reflect that the % complete is 100. Are there any tools available to do this?
>>
>> Also at some point I must figure out why its happening :)
>>
>> Cheers,
>> Tom
>>
>>
>>
>> --
>> -Sheryl
>>
>
>
> ---------------------------------------------------------------------
> CONFIDENTIALITY NOTICE: This e-mail message, including any attachments,
> is for the sole use of the intended recipient(s) and may contain confidential
> or legally privileged information. Any unauthorized review, use, disclosure
> or distribution is prohibited. If you are not the intended recipient, please
> contact the sender by reply e-mail and destroy all copies of this original message.
>
> ---------------------------------------------------------------------
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: How can I update a filemanager's catalogue?
Posted by "Nguyen, Ricky" <rn...@chla.usc.edu>.
Awhile back, I wrote a tiny Java main program to edit a Lucene catalog (attempting to clean our catalog), but discovered that once you write out v3.x, you can't read it back in with OODT.
I believe OODT uses Lucene 2.0 and latest Luke uses 3.x. If Luke has the option to pick your Lucene format version, you may be good to go.
-Ricky
On Jul 16, 2012, at 11:32 AM, Thomas Bennett wrote:
Hey Sheryl and Cameron,
I have downloaded Luke with the intention of trying it out. Thanks for the warning Sheryl. I'll give it a try unless Ricky advises otherwise.
Yeah, the files are all okay i.e. have size and content, so I suspect a bug of some sort.
I'm planing to take a look to see how file transfer status is updated. Since I'm using the ingest with the InPlaceDataTrasfer (I think that's the option for the crawler) it may have introduces an interesting aspect since the file is not actually transferred :)
Cheers,
Tom
On 16 July 2012 17:40, Sheryl John <sh...@gmail.com>> wrote:
Hi Cameron,
Were you able to edit the filemgr lucene catalog?
I think if you update an index with Luke it would write out a version that can't be read by an older Solr/Lucene version.
I remember Ricky trying to edit the filemgr catalog with Luke and faced the above issue.
On Mon, Jul 16, 2012 at 8:09 AM, Cameron Goodale <si...@gmail.com>> wrote:
Tom,
You could use lucene's Luke tool to query and edit the catalog. Paul and I discovered a scripting plugin within Luke that can execute javascript to manipulate the catalog.
I hope that is a good lead for you.
Cameron
On Jul 16, 2012 8:26 AM, "Thomas Bennett" <lm...@gmail.com>> wrote:
Hi,
I've recently created a Lucene file manager catalog. I cataloged about 2000 files "in place", but have encountered a problem.
From the fmbrowser some of the files are listed as being:
Product Transfer Status Percent Complete Received Time References Metadata
1287587805.h5 RECEIVED 100% UNKNOWN View References View Metadata
1287353769.h5 RECEIVED 0% UNKNOWN View References View Metadata
1287355996.h5 RECEIVED 0% UNKNOWN View References View Metadata
1287425397.h5 RECEIVED 100% UNKNOWN View References View Metadata
The problem lies with the two products that show % complete being 0%.
When I use fmprod to download the product:
wget http://..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e<http://kat-archive.kat.ac.za:8983/fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e>
I get:
HTTP request sent, awaiting response... 500 Internal Server Error
2012-07-14 20:20:38 ERROR 500: Internal Server Error.
So I'd like to "edit" the lucene catalog to update this piece of metadata to reflect that the % complete is 100. Are there any tools available to do this?
Also at some point I must figure out why its happening :)
Cheers,
Tom
--
-Sheryl
---------------------------------------------------------------------
CONFIDENTIALITY NOTICE: This e-mail message, including any attachments,
is for the sole use of the intended recipient(s) and may contain confidential
or legally privileged information. Any unauthorized review, use, disclosure
or distribution is prohibited. If you are not the intended recipient, please
contact the sender by reply e-mail and destroy all copies of this original message.
---------------------------------------------------------------------
Re: How can I update a filemanager's catalogue?
Posted by "Nguyen, Ricky" <rn...@chla.usc.edu>.
Awhile back, I wrote a tiny Java main program to edit a Lucene catalog (attempting to clean our catalog), but discovered that once you write out v3.x, you can't read it back in with OODT.
I believe OODT uses Lucene 2.0 and latest Luke uses 3.x. If Luke has the option to pick your Lucene format version, you may be good to go.
-Ricky
On Jul 16, 2012, at 11:32 AM, Thomas Bennett wrote:
Hey Sheryl and Cameron,
I have downloaded Luke with the intention of trying it out. Thanks for the warning Sheryl. I'll give it a try unless Ricky advises otherwise.
Yeah, the files are all okay i.e. have size and content, so I suspect a bug of some sort.
I'm planing to take a look to see how file transfer status is updated. Since I'm using the ingest with the InPlaceDataTrasfer (I think that's the option for the crawler) it may have introduces an interesting aspect since the file is not actually transferred :)
Cheers,
Tom
On 16 July 2012 17:40, Sheryl John <sh...@gmail.com>> wrote:
Hi Cameron,
Were you able to edit the filemgr lucene catalog?
I think if you update an index with Luke it would write out a version that can't be read by an older Solr/Lucene version.
I remember Ricky trying to edit the filemgr catalog with Luke and faced the above issue.
On Mon, Jul 16, 2012 at 8:09 AM, Cameron Goodale <si...@gmail.com>> wrote:
Tom,
You could use lucene's Luke tool to query and edit the catalog. Paul and I discovered a scripting plugin within Luke that can execute javascript to manipulate the catalog.
I hope that is a good lead for you.
Cameron
On Jul 16, 2012 8:26 AM, "Thomas Bennett" <lm...@gmail.com>> wrote:
Hi,
I've recently created a Lucene file manager catalog. I cataloged about 2000 files "in place", but have encountered a problem.
From the fmbrowser some of the files are listed as being:
Product Transfer Status Percent Complete Received Time References Metadata
1287587805.h5 RECEIVED 100% UNKNOWN View References View Metadata
1287353769.h5 RECEIVED 0% UNKNOWN View References View Metadata
1287355996.h5 RECEIVED 0% UNKNOWN View References View Metadata
1287425397.h5 RECEIVED 100% UNKNOWN View References View Metadata
The problem lies with the two products that show % complete being 0%.
When I use fmprod to download the product:
wget http://..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e<http://kat-archive.kat.ac.za:8983/fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e>
I get:
HTTP request sent, awaiting response... 500 Internal Server Error
2012-07-14 20:20:38 ERROR 500: Internal Server Error.
So I'd like to "edit" the lucene catalog to update this piece of metadata to reflect that the % complete is 100. Are there any tools available to do this?
Also at some point I must figure out why its happening :)
Cheers,
Tom
--
-Sheryl
---------------------------------------------------------------------
CONFIDENTIALITY NOTICE: This e-mail message, including any attachments,
is for the sole use of the intended recipient(s) and may contain confidential
or legally privileged information. Any unauthorized review, use, disclosure
or distribution is prohibited. If you are not the intended recipient, please
contact the sender by reply e-mail and destroy all copies of this original message.
---------------------------------------------------------------------
Re: How can I update a filemanager's catalogue?
Posted by Thomas Bennett <lm...@gmail.com>.
Hey Sheryl and Cameron,
I have downloaded Luke with the intention of trying it out. Thanks for the
warning Sheryl. I'll give it a try unless Ricky advises otherwise.
Yeah, the files are all okay i.e. have size and content, so I suspect a bug
of some sort.
I'm planing to take a look to see how file transfer status is updated.
Since I'm using the ingest with the InPlaceDataTrasfer (I think that's the
option for the crawler) it may have introduces an interesting aspect since
the file is not actually transferred :)
Cheers,
Tom
On 16 July 2012 17:40, Sheryl John <sh...@gmail.com> wrote:
> Hi Cameron,
>
> Were you able to edit the filemgr lucene catalog?
> I think if you update an index with Luke it would write out a version that
> can't be read by an older Solr/Lucene version.
> I remember Ricky trying to edit the filemgr catalog with Luke and faced
> the above issue.
>
>
> On Mon, Jul 16, 2012 at 8:09 AM, Cameron Goodale <si...@gmail.com>wrote:
>
>> Tom,
>>
>> You could use lucene's Luke tool to query and edit the catalog. Paul and
>> I discovered a scripting plugin within Luke that can execute javascript to
>> manipulate the catalog.
>>
>> I hope that is a good lead for you.
>>
>> Cameron
>> On Jul 16, 2012 8:26 AM, "Thomas Bennett" <lm...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I've recently created a Lucene file manager catalog. I cataloged about
>>> 2000 files "in place", but have encountered a problem.
>>>
>>> From the fmbrowser some of the files are listed as being:
>>>
>>> Product Transfer Status Percent Complete Received Time
>>> References Metadata
>>> 1287587805.h5 RECEIVED 100% UNKNOWN View References
>>> View Metadata
>>> 1287353769.h5 RECEIVED *0%* UNKNOWN View
>>> References View Metadata
>>> 1287355996.h5 RECEIVED *0%* UNKNOWN View
>>> References View Metadata
>>> 1287425397.h5 RECEIVED 100% UNKNOWN View References
>>> View Metadata
>>>
>>> The problem lies with the two products that show % complete being 0%.
>>>
>>> When I use fmprod to download the product:
>>>
>>> wget
>>> http://..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e<http://kat-archive.kat.ac.za:8983/fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e>
>>>
>>> I get:
>>> HTTP request sent, awaiting response... 500 Internal Server Error
>>> 2012-07-14 20:20:38 ERROR 500: Internal Server Error.
>>>
>>> So I'd like to "edit" the lucene catalog to update this piece of
>>> metadata to reflect that the % complete is 100. Are there any tools
>>> available to do this?
>>>
>>> Also at some point I must figure out why its happening :)
>>>
>>> Cheers,
>>> Tom
>>>
>>
>
>
> --
> -Sheryl
>
Re: How can I update a filemanager's catalogue?
Posted by Thomas Bennett <lm...@gmail.com>.
Hey Sheryl and Cameron,
I have downloaded Luke with the intention of trying it out. Thanks for the
warning Sheryl. I'll give it a try unless Ricky advises otherwise.
Yeah, the files are all okay i.e. have size and content, so I suspect a bug
of some sort.
I'm planing to take a look to see how file transfer status is updated.
Since I'm using the ingest with the InPlaceDataTrasfer (I think that's the
option for the crawler) it may have introduces an interesting aspect since
the file is not actually transferred :)
Cheers,
Tom
On 16 July 2012 17:40, Sheryl John <sh...@gmail.com> wrote:
> Hi Cameron,
>
> Were you able to edit the filemgr lucene catalog?
> I think if you update an index with Luke it would write out a version that
> can't be read by an older Solr/Lucene version.
> I remember Ricky trying to edit the filemgr catalog with Luke and faced
> the above issue.
>
>
> On Mon, Jul 16, 2012 at 8:09 AM, Cameron Goodale <si...@gmail.com>wrote:
>
>> Tom,
>>
>> You could use lucene's Luke tool to query and edit the catalog. Paul and
>> I discovered a scripting plugin within Luke that can execute javascript to
>> manipulate the catalog.
>>
>> I hope that is a good lead for you.
>>
>> Cameron
>> On Jul 16, 2012 8:26 AM, "Thomas Bennett" <lm...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I've recently created a Lucene file manager catalog. I cataloged about
>>> 2000 files "in place", but have encountered a problem.
>>>
>>> From the fmbrowser some of the files are listed as being:
>>>
>>> Product Transfer Status Percent Complete Received Time
>>> References Metadata
>>> 1287587805.h5 RECEIVED 100% UNKNOWN View References
>>> View Metadata
>>> 1287353769.h5 RECEIVED *0%* UNKNOWN View
>>> References View Metadata
>>> 1287355996.h5 RECEIVED *0%* UNKNOWN View
>>> References View Metadata
>>> 1287425397.h5 RECEIVED 100% UNKNOWN View References
>>> View Metadata
>>>
>>> The problem lies with the two products that show % complete being 0%.
>>>
>>> When I use fmprod to download the product:
>>>
>>> wget
>>> http://..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e<http://kat-archive.kat.ac.za:8983/fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e>
>>>
>>> I get:
>>> HTTP request sent, awaiting response... 500 Internal Server Error
>>> 2012-07-14 20:20:38 ERROR 500: Internal Server Error.
>>>
>>> So I'd like to "edit" the lucene catalog to update this piece of
>>> metadata to reflect that the % complete is 100. Are there any tools
>>> available to do this?
>>>
>>> Also at some point I must figure out why its happening :)
>>>
>>> Cheers,
>>> Tom
>>>
>>
>
>
> --
> -Sheryl
>
Re: How can I update a filemanager's catalogue?
Posted by Cameron Goodale <si...@gmail.com>.
Sheryl,
Thanks for the warning. Paul and I stumbled onto the Luke scripting feature
but didn't use them.
Lucene catalog cleansing is something we will be addressing for the snow
project. If anything useful comes out of it, we'll be sure to share.
Cameron
On Jul 16, 2012 9:40 AM, "Sheryl John" <sh...@gmail.com> wrote:
> Hi Cameron,
>
> Were you able to edit the filemgr lucene catalog?
> I think if you update an index with Luke it would write out a version that
> can't be read by an older Solr/Lucene version.
> I remember Ricky trying to edit the filemgr catalog with Luke and faced
> the above issue.
>
> On Mon, Jul 16, 2012 at 8:09 AM, Cameron Goodale <si...@gmail.com>wrote:
>
>> Tom,
>>
>> You could use lucene's Luke tool to query and edit the catalog. Paul and
>> I discovered a scripting plugin within Luke that can execute javascript to
>> manipulate the catalog.
>>
>> I hope that is a good lead for you.
>>
>> Cameron
>> On Jul 16, 2012 8:26 AM, "Thomas Bennett" <lm...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I've recently created a Lucene file manager catalog. I cataloged about
>>> 2000 files "in place", but have encountered a problem.
>>>
>>> From the fmbrowser some of the files are listed as being:
>>>
>>> Product Transfer Status Percent Complete Received Time
>>> References Metadata
>>> 1287587805.h5 RECEIVED 100% UNKNOWN View References
>>> View Metadata
>>> 1287353769.h5 RECEIVED *0%* UNKNOWN View
>>> References View Metadata
>>> 1287355996.h5 RECEIVED *0%* UNKNOWN View
>>> References View Metadata
>>> 1287425397.h5 RECEIVED 100% UNKNOWN View References
>>> View Metadata
>>>
>>> The problem lies with the two products that show % complete being 0%.
>>>
>>> When I use fmprod to download the product:
>>>
>>> wget
>>> http://..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e<http://kat-archive.kat.ac.za:8983/fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e>
>>>
>>> I get:
>>> HTTP request sent, awaiting response... 500 Internal Server Error
>>> 2012-07-14 20:20:38 ERROR 500: Internal Server Error.
>>>
>>> So I'd like to "edit" the lucene catalog to update this piece of
>>> metadata to reflect that the % complete is 100. Are there any tools
>>> available to do this?
>>>
>>> Also at some point I must figure out why its happening :)
>>>
>>> Cheers,
>>> Tom
>>>
>>
>
>
> --
> -Sheryl
>
Re: How can I update a filemanager's catalogue?
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Tom,
On Jul 17, 2012, at 7:57 AM, Thomas Bennett wrote:
> Hi everyone,
>
> Thanks for the responses. I always appreciate the fact that post a question/problem on the forum and the next morning there are stack of replies.
>
> +1 to everyone!
Thanks dude!
>
> As it turns out the problem was entirely mine :-)
Well not entirely! :)
Read on below:
>
> However, I've discovered how the file manager works out the percentage file transferred and why it thinks the file Percentage Complete was 0%.
>
> The magic all happens with the following piece of code:
>
> private long getBytesTransferred(Reference r) {
> File destFile = null;
> try {
> destFile = new File(new URI(r.getDataStoreReference()));
> long bytesTransferred = destFile.length();
> return bytesTransferred;
> } catch (URISyntaxException e) {
> LOG.log(Level.WARNING,
> "URISyntaxException when checking size of destFile: ["
> + r.getDataStoreReference() + "]: Message: "
> + e.getMessage());
> return -1L;
> }
> }
>
> So the file transfer size is worked out on the fly - its not stored in the catalog.
>
> The origin of the weirdness relates to my Versioner.
>
> My versioner stores files to file:/var/kat/archive/data/comm/[StartDate]/1234567890.h5
>
> My original code (pre-OODT) stored the data in the same way. However, for OODT I store my dates in UTC, and thus there is a 2 hour difference between my preOODT and OODT implementations, so for my product 1287353769.h5 the references are:
>
> origReference "file:/var/kat/archive/data/comm/2010/10/18/1287353769.h5" (id=115)
> dataStoreReference "file:/var/kat/archive/data/comm/2010/10/17/1287353769.h5" (id=112)
>
> Since I used the InPlaceDataTransfer the data reference was updated but the file was not moved.
>
> So files captured between midnight and 2am will now have a new reference to the time shifted directory (e.g. moved from the 18'th to the 17'th)
>
> When the %complete is calculated it uses the dataStoreReference.
>
> Is it possible for the file manager to know that the InPlaceDataTransfer has been used and that the reference should not change? I guess this would not be possible.
Yep it's possible. You can use the InPlaceVersioner, and extend it, or extend MetadataBasedVersioner and sub-class to do something similar to the InPlaceVersioner.
Does that make sense?
Cheers,
Chris
>
> On 17 July 2012 05:29, Sheryl John <sh...@gmail.com> wrote:
> > On Jul 16, 2012 8:26 AM, "Thomas Bennett" <lm...@gmail.com> wrote:
> > Hi,
> >
> > I've recently created a Lucene file manager catalog. I cataloged about 2000 files "in place", but have encountered a problem.
> >
> > From the fmbrowser some of the files are listed as being:
> >
> > Product Transfer Status Percent Complete Received Time References Metadata
> > 1287587805.h5 RECEIVED 100% UNKNOWN View References View Metadata
> > 1287353769.h5 RECEIVED 0% UNKNOWN View References View Metadata
> > 1287355996.h5 RECEIVED 0% UNKNOWN View References View Metadata
> > 1287425397.h5 RECEIVED 100% UNKNOWN View References View Metadata
> >
> > The problem lies with the two products that show % complete being 0%.
> >
> > When I use fmprod to download the product:
> >
> > wget http://..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e
> >
> > I get:
> > HTTP request sent, awaiting response... 500 Internal Server Error
> > 2012-07-14 20:20:38 ERROR 500: Internal Server Error.
> >
> > So I'd like to "edit" the lucene catalog to update this piece of metadata to reflect that the % complete is 100. Are there any tools available to do this?
> >
> > Also at some point I must figure out why its happening :)
> >
> > Cheers,
> > Tom
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
Phone: +1 (818) 354-8810
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: How can I update a filemanager's catalogue?
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Tom,
On Jul 17, 2012, at 7:57 AM, Thomas Bennett wrote:
> Hi everyone,
>
> Thanks for the responses. I always appreciate the fact that post a question/problem on the forum and the next morning there are stack of replies.
>
> +1 to everyone!
Thanks dude!
>
> As it turns out the problem was entirely mine :-)
Well not entirely! :)
Read on below:
>
> However, I've discovered how the file manager works out the percentage file transferred and why it thinks the file Percentage Complete was 0%.
>
> The magic all happens with the following piece of code:
>
> private long getBytesTransferred(Reference r) {
> File destFile = null;
> try {
> destFile = new File(new URI(r.getDataStoreReference()));
> long bytesTransferred = destFile.length();
> return bytesTransferred;
> } catch (URISyntaxException e) {
> LOG.log(Level.WARNING,
> "URISyntaxException when checking size of destFile: ["
> + r.getDataStoreReference() + "]: Message: "
> + e.getMessage());
> return -1L;
> }
> }
>
> So the file transfer size is worked out on the fly - its not stored in the catalog.
>
> The origin of the weirdness relates to my Versioner.
>
> My versioner stores files to file:/var/kat/archive/data/comm/[StartDate]/1234567890.h5
>
> My original code (pre-OODT) stored the data in the same way. However, for OODT I store my dates in UTC, and thus there is a 2 hour difference between my preOODT and OODT implementations, so for my product 1287353769.h5 the references are:
>
> origReference "file:/var/kat/archive/data/comm/2010/10/18/1287353769.h5" (id=115)
> dataStoreReference "file:/var/kat/archive/data/comm/2010/10/17/1287353769.h5" (id=112)
>
> Since I used the InPlaceDataTransfer the data reference was updated but the file was not moved.
>
> So files captured between midnight and 2am will now have a new reference to the time shifted directory (e.g. moved from the 18'th to the 17'th)
>
> When the %complete is calculated it uses the dataStoreReference.
>
> Is it possible for the file manager to know that the InPlaceDataTransfer has been used and that the reference should not change? I guess this would not be possible.
Yep it's possible. You can use the InPlaceVersioner, and extend it, or extend MetadataBasedVersioner and sub-class to do something similar to the InPlaceVersioner.
Does that make sense?
Cheers,
Chris
>
> On 17 July 2012 05:29, Sheryl John <sh...@gmail.com> wrote:
> > On Jul 16, 2012 8:26 AM, "Thomas Bennett" <lm...@gmail.com> wrote:
> > Hi,
> >
> > I've recently created a Lucene file manager catalog. I cataloged about 2000 files "in place", but have encountered a problem.
> >
> > From the fmbrowser some of the files are listed as being:
> >
> > Product Transfer Status Percent Complete Received Time References Metadata
> > 1287587805.h5 RECEIVED 100% UNKNOWN View References View Metadata
> > 1287353769.h5 RECEIVED 0% UNKNOWN View References View Metadata
> > 1287355996.h5 RECEIVED 0% UNKNOWN View References View Metadata
> > 1287425397.h5 RECEIVED 100% UNKNOWN View References View Metadata
> >
> > The problem lies with the two products that show % complete being 0%.
> >
> > When I use fmprod to download the product:
> >
> > wget http://..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e
> >
> > I get:
> > HTTP request sent, awaiting response... 500 Internal Server Error
> > 2012-07-14 20:20:38 ERROR 500: Internal Server Error.
> >
> > So I'd like to "edit" the lucene catalog to update this piece of metadata to reflect that the % complete is 100. Are there any tools available to do this?
> >
> > Also at some point I must figure out why its happening :)
> >
> > Cheers,
> > Tom
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
Phone: +1 (818) 354-8810
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: How can I update a filemanager's catalogue?
Posted by Thomas Bennett <lm...@gmail.com>.
Hi everyone,
Thanks for the responses. I always appreciate the fact that post a
question/problem on the forum and the next morning there are stack of
replies.
+1 to everyone!
As it turns out the problem was entirely mine :-)
However, I've discovered how the file manager works out the percentage
file transferred and why it thinks the file Percentage Complete was 0%.
The magic all happens with the following piece of code:
private long getBytesTransferred(Reference r) {
File destFile = null;
try {
destFile = new File(new URI(r.getDataStoreReference()));
* long bytesTransferred = destFile.length();*
return bytesTransferred;
} catch (URISyntaxException e) {
LOG.log(Level.WARNING,
"URISyntaxException when checking size of destFile: ["
+ r.getDataStoreReference() + "]: Message: "
+ e.getMessage());
return -1L;
}
}
So the file transfer size is worked out on the fly - its not stored in the
catalog.
The origin of the weirdness relates to my Versioner.
My versioner stores files to
file:/var/kat/archive/data/comm/[StartDate]/1234567890.h5
My original code (pre-OODT) stored the data in the same way. However, for
OODT I store my dates in UTC, and thus there is a 2 hour difference between
my preOODT and OODT implementations, so for my product 1287353769.h5 the
references are:
origReference "file:/var/kat/archive/data/comm/2010/10/*18*/1287353769.h5"
(id=115)
dataStoreReference "file:/var/kat/archive/data/comm/2010/10/*17*/1287353769.h5"
(id=112)
Since I used the InPlaceDataTransfer the data reference was updated but the
file was not moved.
So files captured between midnight and 2am will now have a new reference to
the time shifted directory (e.g. moved from the 18'th to the 17'th)
When the %complete is calculated it uses the dataStoreReference.
Is it possible for the file manager to know that the InPlaceDataTransfer
has been used and that the reference should not change? I guess this would
not be possible.
Cheers,
Tom
On 17 July 2012 05:29, Sheryl John <sh...@gmail.com> wrote:
> > On Jul 16, 2012 8:26 AM, "Thomas Bennett" <lm...@gmail.com> wrote:
>> > Hi,
>> >
>> > I've recently created a Lucene file manager catalog. I cataloged about
>> 2000 files "in place", but have encountered a problem.
>> >
>> > From the fmbrowser some of the files are listed as being:
>> >
>> > Product Transfer Status Percent Complete Received
>> Time References Metadata
>> > 1287587805.h5 RECEIVED 100% UNKNOWN
>> View References View Metadata
>> > 1287353769.h5 RECEIVED 0% UNKNOWN
>> View References View Metadata
>> > 1287355996.h5 RECEIVED 0% UNKNOWN
>> View References View Metadata
>> > 1287425397.h5 RECEIVED 100% UNKNOWN
>> View References View Metadata
>> >
>> > The problem lies with the two products that show % complete being 0%.
>> >
>> > When I use fmprod to download the product:
>> >
>> > wget http://
>> ..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e
>> >
>> > I get:
>> > HTTP request sent, awaiting response... 500 Internal Server Error
>> > 2012-07-14 20:20:38 ERROR 500: Internal Server Error.
>> >
>> > So I'd like to "edit" the lucene catalog to update this piece of
>> metadata to reflect that the % complete is 100. Are there any tools
>> available to do this?
>> >
>> > Also at some point I must figure out why its happening :)
>> >
>> > Cheers,
>> > Tom
>>
>
Re: How can I update a filemanager's catalogue?
Posted by Thomas Bennett <lm...@gmail.com>.
Hi everyone,
Thanks for the responses. I always appreciate the fact that post a
question/problem on the forum and the next morning there are stack of
replies.
+1 to everyone!
As it turns out the problem was entirely mine :-)
However, I've discovered how the file manager works out the percentage
file transferred and why it thinks the file Percentage Complete was 0%.
The magic all happens with the following piece of code:
private long getBytesTransferred(Reference r) {
File destFile = null;
try {
destFile = new File(new URI(r.getDataStoreReference()));
* long bytesTransferred = destFile.length();*
return bytesTransferred;
} catch (URISyntaxException e) {
LOG.log(Level.WARNING,
"URISyntaxException when checking size of destFile: ["
+ r.getDataStoreReference() + "]: Message: "
+ e.getMessage());
return -1L;
}
}
So the file transfer size is worked out on the fly - its not stored in the
catalog.
The origin of the weirdness relates to my Versioner.
My versioner stores files to
file:/var/kat/archive/data/comm/[StartDate]/1234567890.h5
My original code (pre-OODT) stored the data in the same way. However, for
OODT I store my dates in UTC, and thus there is a 2 hour difference between
my preOODT and OODT implementations, so for my product 1287353769.h5 the
references are:
origReference "file:/var/kat/archive/data/comm/2010/10/*18*/1287353769.h5"
(id=115)
dataStoreReference "file:/var/kat/archive/data/comm/2010/10/*17*/1287353769.h5"
(id=112)
Since I used the InPlaceDataTransfer the data reference was updated but the
file was not moved.
So files captured between midnight and 2am will now have a new reference to
the time shifted directory (e.g. moved from the 18'th to the 17'th)
When the %complete is calculated it uses the dataStoreReference.
Is it possible for the file manager to know that the InPlaceDataTransfer
has been used and that the reference should not change? I guess this would
not be possible.
Cheers,
Tom
On 17 July 2012 05:29, Sheryl John <sh...@gmail.com> wrote:
> > On Jul 16, 2012 8:26 AM, "Thomas Bennett" <lm...@gmail.com> wrote:
>> > Hi,
>> >
>> > I've recently created a Lucene file manager catalog. I cataloged about
>> 2000 files "in place", but have encountered a problem.
>> >
>> > From the fmbrowser some of the files are listed as being:
>> >
>> > Product Transfer Status Percent Complete Received
>> Time References Metadata
>> > 1287587805.h5 RECEIVED 100% UNKNOWN
>> View References View Metadata
>> > 1287353769.h5 RECEIVED 0% UNKNOWN
>> View References View Metadata
>> > 1287355996.h5 RECEIVED 0% UNKNOWN
>> View References View Metadata
>> > 1287425397.h5 RECEIVED 100% UNKNOWN
>> View References View Metadata
>> >
>> > The problem lies with the two products that show % complete being 0%.
>> >
>> > When I use fmprod to download the product:
>> >
>> > wget http://
>> ..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e
>> >
>> > I get:
>> > HTTP request sent, awaiting response... 500 Internal Server Error
>> > 2012-07-14 20:20:38 ERROR 500: Internal Server Error.
>> >
>> > So I'd like to "edit" the lucene catalog to update this piece of
>> metadata to reflect that the % complete is 100. Are there any tools
>> available to do this?
>> >
>> > Also at some point I must figure out why its happening :)
>> >
>> > Cheers,
>> > Tom
>>
>
Re: How can I update a filemanager's catalogue?
Posted by Sheryl John <sh...@gmail.com>.
Thanks Chris!
And +1 for upgrading to a newer version of Lucene for 0.5.
On Mon, Jul 16, 2012 at 4:14 PM, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Hey Sheryl,
>
> On Jul 16, 2012, at 10:40 AM, Sheryl John wrote:
>
> > Hi Cameron,
> >
> > Were you able to edit the filemgr lucene catalog?
> > I think if you update an index with Luke it would write out a version
> that can't be read by an older Solr/Lucene version.
>
> That's possible, yep, depending on the Luke version you use. If you
> use a newer version it will automatically upgrade the catalog to something
> that isn't compatible with lucene-core-2.0.0.jar, the archaic version of
> Lucene
> that we depend on.
>
> However, if you find something like lukeall-0.6.2 or 0.7.0 .jar then I
> think
> it will work fine.
>
> > I remember Ricky trying to edit the filemgr catalog with Luke and faced
> the above issue.
>
> Yep see above.
>
> Cheers,
> Chris
>
> >
> > On Mon, Jul 16, 2012 at 8:09 AM, Cameron Goodale <si...@gmail.com>
> wrote:
> > Tom,
> >
> > You could use lucene's Luke tool to query and edit the catalog. Paul and
> I discovered a scripting plugin within Luke that can execute javascript to
> manipulate the catalog.
> >
> > I hope that is a good lead for you.
> >
> > Cameron
> >
> > On Jul 16, 2012 8:26 AM, "Thomas Bennett" <lm...@gmail.com> wrote:
> > Hi,
> >
> > I've recently created a Lucene file manager catalog. I cataloged about
> 2000 files "in place", but have encountered a problem.
> >
> > From the fmbrowser some of the files are listed as being:
> >
> > Product Transfer Status Percent Complete Received
> Time References Metadata
> > 1287587805.h5 RECEIVED 100% UNKNOWN
> View References View Metadata
> > 1287353769.h5 RECEIVED 0% UNKNOWN
> View References View Metadata
> > 1287355996.h5 RECEIVED 0% UNKNOWN
> View References View Metadata
> > 1287425397.h5 RECEIVED 100% UNKNOWN
> View References View Metadata
> >
> > The problem lies with the two products that show % complete being 0%.
> >
> > When I use fmprod to download the product:
> >
> > wget http://
> ..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e
> >
> > I get:
> > HTTP request sent, awaiting response... 500 Internal Server Error
> > 2012-07-14 20:20:38 ERROR 500: Internal Server Error.
> >
> > So I'd like to "edit" the lucene catalog to update this piece of
> metadata to reflect that the % complete is 100. Are there any tools
> available to do this?
> >
> > Also at some point I must figure out why its happening :)
> >
> > Cheers,
> > Tom
> >
> >
> >
> > --
> > -Sheryl
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
--
-Sheryl
Re: How can I update a filemanager's catalogue?
Posted by Sheryl John <sh...@gmail.com>.
Thanks Chris!
And +1 for upgrading to a newer version of Lucene for 0.5.
On Mon, Jul 16, 2012 at 4:14 PM, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Hey Sheryl,
>
> On Jul 16, 2012, at 10:40 AM, Sheryl John wrote:
>
> > Hi Cameron,
> >
> > Were you able to edit the filemgr lucene catalog?
> > I think if you update an index with Luke it would write out a version
> that can't be read by an older Solr/Lucene version.
>
> That's possible, yep, depending on the Luke version you use. If you
> use a newer version it will automatically upgrade the catalog to something
> that isn't compatible with lucene-core-2.0.0.jar, the archaic version of
> Lucene
> that we depend on.
>
> However, if you find something like lukeall-0.6.2 or 0.7.0 .jar then I
> think
> it will work fine.
>
> > I remember Ricky trying to edit the filemgr catalog with Luke and faced
> the above issue.
>
> Yep see above.
>
> Cheers,
> Chris
>
> >
> > On Mon, Jul 16, 2012 at 8:09 AM, Cameron Goodale <si...@gmail.com>
> wrote:
> > Tom,
> >
> > You could use lucene's Luke tool to query and edit the catalog. Paul and
> I discovered a scripting plugin within Luke that can execute javascript to
> manipulate the catalog.
> >
> > I hope that is a good lead for you.
> >
> > Cameron
> >
> > On Jul 16, 2012 8:26 AM, "Thomas Bennett" <lm...@gmail.com> wrote:
> > Hi,
> >
> > I've recently created a Lucene file manager catalog. I cataloged about
> 2000 files "in place", but have encountered a problem.
> >
> > From the fmbrowser some of the files are listed as being:
> >
> > Product Transfer Status Percent Complete Received
> Time References Metadata
> > 1287587805.h5 RECEIVED 100% UNKNOWN
> View References View Metadata
> > 1287353769.h5 RECEIVED 0% UNKNOWN
> View References View Metadata
> > 1287355996.h5 RECEIVED 0% UNKNOWN
> View References View Metadata
> > 1287425397.h5 RECEIVED 100% UNKNOWN
> View References View Metadata
> >
> > The problem lies with the two products that show % complete being 0%.
> >
> > When I use fmprod to download the product:
> >
> > wget http://
> ..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e
> >
> > I get:
> > HTTP request sent, awaiting response... 500 Internal Server Error
> > 2012-07-14 20:20:38 ERROR 500: Internal Server Error.
> >
> > So I'd like to "edit" the lucene catalog to update this piece of
> metadata to reflect that the % complete is 100. Are there any tools
> available to do this?
> >
> > Also at some point I must figure out why its happening :)
> >
> > Cheers,
> > Tom
> >
> >
> >
> > --
> > -Sheryl
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
--
-Sheryl
Re: How can I update a filemanager's catalogue?
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Sheryl,
On Jul 16, 2012, at 10:40 AM, Sheryl John wrote:
> Hi Cameron,
>
> Were you able to edit the filemgr lucene catalog?
> I think if you update an index with Luke it would write out a version that can't be read by an older Solr/Lucene version.
That's possible, yep, depending on the Luke version you use. If you
use a newer version it will automatically upgrade the catalog to something
that isn't compatible with lucene-core-2.0.0.jar, the archaic version of Lucene
that we depend on.
However, if you find something like lukeall-0.6.2 or 0.7.0 .jar then I think
it will work fine.
> I remember Ricky trying to edit the filemgr catalog with Luke and faced the above issue.
Yep see above.
Cheers,
Chris
>
> On Mon, Jul 16, 2012 at 8:09 AM, Cameron Goodale <si...@gmail.com> wrote:
> Tom,
>
> You could use lucene's Luke tool to query and edit the catalog. Paul and I discovered a scripting plugin within Luke that can execute javascript to manipulate the catalog.
>
> I hope that is a good lead for you.
>
> Cameron
>
> On Jul 16, 2012 8:26 AM, "Thomas Bennett" <lm...@gmail.com> wrote:
> Hi,
>
> I've recently created a Lucene file manager catalog. I cataloged about 2000 files "in place", but have encountered a problem.
>
> From the fmbrowser some of the files are listed as being:
>
> Product Transfer Status Percent Complete Received Time References Metadata
> 1287587805.h5 RECEIVED 100% UNKNOWN View References View Metadata
> 1287353769.h5 RECEIVED 0% UNKNOWN View References View Metadata
> 1287355996.h5 RECEIVED 0% UNKNOWN View References View Metadata
> 1287425397.h5 RECEIVED 100% UNKNOWN View References View Metadata
>
> The problem lies with the two products that show % complete being 0%.
>
> When I use fmprod to download the product:
>
> wget http://..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e
>
> I get:
> HTTP request sent, awaiting response... 500 Internal Server Error
> 2012-07-14 20:20:38 ERROR 500: Internal Server Error.
>
> So I'd like to "edit" the lucene catalog to update this piece of metadata to reflect that the % complete is 100. Are there any tools available to do this?
>
> Also at some point I must figure out why its happening :)
>
> Cheers,
> Tom
>
>
>
> --
> -Sheryl
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: How can I update a filemanager's catalogue?
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Sheryl,
On Jul 16, 2012, at 10:40 AM, Sheryl John wrote:
> Hi Cameron,
>
> Were you able to edit the filemgr lucene catalog?
> I think if you update an index with Luke it would write out a version that can't be read by an older Solr/Lucene version.
That's possible, yep, depending on the Luke version you use. If you
use a newer version it will automatically upgrade the catalog to something
that isn't compatible with lucene-core-2.0.0.jar, the archaic version of Lucene
that we depend on.
However, if you find something like lukeall-0.6.2 or 0.7.0 .jar then I think
it will work fine.
> I remember Ricky trying to edit the filemgr catalog with Luke and faced the above issue.
Yep see above.
Cheers,
Chris
>
> On Mon, Jul 16, 2012 at 8:09 AM, Cameron Goodale <si...@gmail.com> wrote:
> Tom,
>
> You could use lucene's Luke tool to query and edit the catalog. Paul and I discovered a scripting plugin within Luke that can execute javascript to manipulate the catalog.
>
> I hope that is a good lead for you.
>
> Cameron
>
> On Jul 16, 2012 8:26 AM, "Thomas Bennett" <lm...@gmail.com> wrote:
> Hi,
>
> I've recently created a Lucene file manager catalog. I cataloged about 2000 files "in place", but have encountered a problem.
>
> From the fmbrowser some of the files are listed as being:
>
> Product Transfer Status Percent Complete Received Time References Metadata
> 1287587805.h5 RECEIVED 100% UNKNOWN View References View Metadata
> 1287353769.h5 RECEIVED 0% UNKNOWN View References View Metadata
> 1287355996.h5 RECEIVED 0% UNKNOWN View References View Metadata
> 1287425397.h5 RECEIVED 100% UNKNOWN View References View Metadata
>
> The problem lies with the two products that show % complete being 0%.
>
> When I use fmprod to download the product:
>
> wget http://..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e
>
> I get:
> HTTP request sent, awaiting response... 500 Internal Server Error
> 2012-07-14 20:20:38 ERROR 500: Internal Server Error.
>
> So I'd like to "edit" the lucene catalog to update this piece of metadata to reflect that the % complete is 100. Are there any tools available to do this?
>
> Also at some point I must figure out why its happening :)
>
> Cheers,
> Tom
>
>
>
> --
> -Sheryl
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: How can I update a filemanager's catalogue?
Posted by Sheryl John <sh...@gmail.com>.
Hi Cameron,
Were you able to edit the filemgr lucene catalog?
I think if you update an index with Luke it would write out a version that
can't be read by an older Solr/Lucene version.
I remember Ricky trying to edit the filemgr catalog with Luke and faced the
above issue.
On Mon, Jul 16, 2012 at 8:09 AM, Cameron Goodale <si...@gmail.com> wrote:
> Tom,
>
> You could use lucene's Luke tool to query and edit the catalog. Paul and I
> discovered a scripting plugin within Luke that can execute javascript to
> manipulate the catalog.
>
> I hope that is a good lead for you.
>
> Cameron
> On Jul 16, 2012 8:26 AM, "Thomas Bennett" <lm...@gmail.com> wrote:
>
>> Hi,
>>
>> I've recently created a Lucene file manager catalog. I cataloged about
>> 2000 files "in place", but have encountered a problem.
>>
>> From the fmbrowser some of the files are listed as being:
>>
>> Product Transfer Status Percent Complete Received Time
>> References Metadata
>> 1287587805.h5 RECEIVED 100% UNKNOWN View References
>> View Metadata
>> 1287353769.h5 RECEIVED *0%* UNKNOWN View
>> References View Metadata
>> 1287355996.h5 RECEIVED *0%* UNKNOWN View
>> References View Metadata
>> 1287425397.h5 RECEIVED 100% UNKNOWN View References
>> View Metadata
>>
>> The problem lies with the two products that show % complete being 0%.
>>
>> When I use fmprod to download the product:
>>
>> wget
>> http://..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e<http://kat-archive.kat.ac.za:8983/fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e>
>>
>> I get:
>> HTTP request sent, awaiting response... 500 Internal Server Error
>> 2012-07-14 20:20:38 ERROR 500: Internal Server Error.
>>
>> So I'd like to "edit" the lucene catalog to update this piece of metadata
>> to reflect that the % complete is 100. Are there any tools available to do
>> this?
>>
>> Also at some point I must figure out why its happening :)
>>
>> Cheers,
>> Tom
>>
>
--
-Sheryl
Re: How can I update a filemanager's catalogue?
Posted by Sheryl John <sh...@gmail.com>.
Hi Cameron,
Were you able to edit the filemgr lucene catalog?
I think if you update an index with Luke it would write out a version that
can't be read by an older Solr/Lucene version.
I remember Ricky trying to edit the filemgr catalog with Luke and faced the
above issue.
On Mon, Jul 16, 2012 at 8:09 AM, Cameron Goodale <si...@gmail.com> wrote:
> Tom,
>
> You could use lucene's Luke tool to query and edit the catalog. Paul and I
> discovered a scripting plugin within Luke that can execute javascript to
> manipulate the catalog.
>
> I hope that is a good lead for you.
>
> Cameron
> On Jul 16, 2012 8:26 AM, "Thomas Bennett" <lm...@gmail.com> wrote:
>
>> Hi,
>>
>> I've recently created a Lucene file manager catalog. I cataloged about
>> 2000 files "in place", but have encountered a problem.
>>
>> From the fmbrowser some of the files are listed as being:
>>
>> Product Transfer Status Percent Complete Received Time
>> References Metadata
>> 1287587805.h5 RECEIVED 100% UNKNOWN View References
>> View Metadata
>> 1287353769.h5 RECEIVED *0%* UNKNOWN View
>> References View Metadata
>> 1287355996.h5 RECEIVED *0%* UNKNOWN View
>> References View Metadata
>> 1287425397.h5 RECEIVED 100% UNKNOWN View References
>> View Metadata
>>
>> The problem lies with the two products that show % complete being 0%.
>>
>> When I use fmprod to download the product:
>>
>> wget
>> http://..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e<http://kat-archive.kat.ac.za:8983/fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e>
>>
>> I get:
>> HTTP request sent, awaiting response... 500 Internal Server Error
>> 2012-07-14 20:20:38 ERROR 500: Internal Server Error.
>>
>> So I'd like to "edit" the lucene catalog to update this piece of metadata
>> to reflect that the % complete is 100. Are there any tools available to do
>> this?
>>
>> Also at some point I must figure out why its happening :)
>>
>> Cheers,
>> Tom
>>
>
--
-Sheryl
Re: How can I update a filemanager's catalogue?
Posted by Cameron Goodale <si...@gmail.com>.
Tom,
You could use lucene's Luke tool to query and edit the catalog. Paul and I
discovered a scripting plugin within Luke that can execute javascript to
manipulate the catalog.
I hope that is a good lead for you.
Cameron
On Jul 16, 2012 8:26 AM, "Thomas Bennett" <lm...@gmail.com> wrote:
> Hi,
>
> I've recently created a Lucene file manager catalog. I cataloged about
> 2000 files "in place", but have encountered a problem.
>
> From the fmbrowser some of the files are listed as being:
>
> Product Transfer Status Percent Complete Received Time
> References Metadata
> 1287587805.h5 RECEIVED 100% UNKNOWN View References
> View Metadata
> 1287353769.h5 RECEIVED *0%* UNKNOWN View
> References View Metadata
> 1287355996.h5 RECEIVED *0%* UNKNOWN View
> References View Metadata
> 1287425397.h5 RECEIVED 100% UNKNOWN View References
> View Metadata
>
> The problem lies with the two products that show % complete being 0%.
>
> When I use fmprod to download the product:
>
> wget
> http://..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e<http://kat-archive.kat.ac.za:8983/fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e>
>
> I get:
> HTTP request sent, awaiting response... 500 Internal Server Error
> 2012-07-14 20:20:38 ERROR 500: Internal Server Error.
>
> So I'd like to "edit" the lucene catalog to update this piece of metadata
> to reflect that the % complete is 100. Are there any tools available to do
> this?
>
> Also at some point I must figure out why its happening :)
>
> Cheers,
> Tom
>
Re: How can I update a filemanager's catalogue?
Posted by Sheryl John <sh...@gmail.com>.
Hi Tom,
I'm not sure if modifying the metadata will help you update the 'percent
complete' status of the file.
Since this is calculated by the filemgr (FileTransferStatus), maybe there
was something wrong with the file size or uploading.
On Mon, Jul 16, 2012 at 7:25 AM, Thomas Bennett <lm...@gmail.com> wrote:
> Hi,
>
> I've recently created a Lucene file manager catalog. I cataloged about
> 2000 files "in place", but have encountered a problem.
>
> From the fmbrowser some of the files are listed as being:
>
> Product Transfer Status Percent Complete Received Time
> References Metadata
> 1287587805.h5 RECEIVED 100% UNKNOWN View References
> View Metadata
> 1287353769.h5 RECEIVED *0%* UNKNOWN View
> References View Metadata
> 1287355996.h5 RECEIVED *0%* UNKNOWN View
> References View Metadata
> 1287425397.h5 RECEIVED 100% UNKNOWN View References
> View Metadata
>
> The problem lies with the two products that show % complete being 0%.
>
> When I use fmprod to download the product:
>
> wget
> http://..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e<http://kat-archive.kat.ac.za:8983/fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e>
>
> I get:
> HTTP request sent, awaiting response... 500 Internal Server Error
> 2012-07-14 20:20:38 ERROR 500: Internal Server Error.
>
> So I'd like to "edit" the lucene catalog to update this piece of metadata
> to reflect that the % complete is 100. Are there any tools available to do
> this?
>
> Also at some point I must figure out why its happening :)
>
> Cheers,
> Tom
>
--
-Sheryl
Re: How can I update a filemanager's catalogue?
Posted by Cameron Goodale <si...@gmail.com>.
Tom,
You could use lucene's Luke tool to query and edit the catalog. Paul and I
discovered a scripting plugin within Luke that can execute javascript to
manipulate the catalog.
I hope that is a good lead for you.
Cameron
On Jul 16, 2012 8:26 AM, "Thomas Bennett" <lm...@gmail.com> wrote:
> Hi,
>
> I've recently created a Lucene file manager catalog. I cataloged about
> 2000 files "in place", but have encountered a problem.
>
> From the fmbrowser some of the files are listed as being:
>
> Product Transfer Status Percent Complete Received Time
> References Metadata
> 1287587805.h5 RECEIVED 100% UNKNOWN View References
> View Metadata
> 1287353769.h5 RECEIVED *0%* UNKNOWN View
> References View Metadata
> 1287355996.h5 RECEIVED *0%* UNKNOWN View
> References View Metadata
> 1287425397.h5 RECEIVED 100% UNKNOWN View References
> View Metadata
>
> The problem lies with the two products that show % complete being 0%.
>
> When I use fmprod to download the product:
>
> wget
> http://..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e<http://kat-archive.kat.ac.za:8983/fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e>
>
> I get:
> HTTP request sent, awaiting response... 500 Internal Server Error
> 2012-07-14 20:20:38 ERROR 500: Internal Server Error.
>
> So I'd like to "edit" the lucene catalog to update this piece of metadata
> to reflect that the % complete is 100. Are there any tools available to do
> this?
>
> Also at some point I must figure out why its happening :)
>
> Cheers,
> Tom
>