You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@archiva.apache.org by Marc Lustig <ml...@marclustig.com> on 2009/09/16 17:19:24 UTC
Re: Archiva 1.2.1 - repos won't get index
Hi,
I gave this another try by doing the following steps.
1. deleting repo/.index and repo/.indexer
2. touching all files and dirs in repo recursively
3. running the repo scan
When we search now for e.g. xmlbeans, we do get a few jar-artifacts that
contain a package-name "xmlbeans". But we do not get the artifact whose name
is xmlbeans.
I have no idea what else to try to make the search-function to work again.
User's here are pretty annoyed about it.
Any suggestion what to do except filing a jira issue?
Deng Ching-2 wrote:
>
> On Thu, Jun 25, 2009 at 7:26 PM, Marc Lustig <ml...@marclustig.com> wrote:
>
>>
>> On our side, the story is continuing like this.
>> In the afternoon, I touch the whole repo and let the scan running and
>> re-create the database.
>> After that, it seems like all (physically) existing artifacts can be
>> found
>> on the Web-UI. Fine.
>> The next morning again hundreds of artifacts are disappeared from the
>> Web-UI
>> (but still existing physically). It's so annoying...
>> I checked the last couple of scans in the log and cannot find exceptions
>> for
>> that Repo concerned.
>> (However I didn't check all the scans since yesterday afternoon.)
>
>
>>
>> Anyway, for some reason, some consumer is removing certain artifacts (or
>> whole groupId's) from the index. I have no clue why.
>>
>> Any suggestion what to do to prevent the removal of artifacts from the
>> index-database?
>>
>> If we disable the "update-db-artifact" consumer, what undesired
>> side-effects
>> may this have?
>>
>>
> Your artifacts won't show up in the webapp browse as this consumer adds
> the
> basic artifact info in the db and that's where the artifacts shown in the
> Browse page is gathered..
>
> You can try disabling these consumers in Database page as they are the
> ones
> doing the cleanup work, but afaik they only clean up the db and index for
> artifacts that are no longer in the filesystem:
> not-present-remove-db-artifact
> not-present-remove-db-project
> not-present-remove-indexed
>
> If the problem still persists, do you mind filing an issue for this in
> jira?
> :)
>
> Thanks,
> Deng
>
>
>>
>> Marc Lustig wrote:
>> >
>> > Indeed,
>> > find [REPOSITORY_DIRECTORY_NAME] | xargs touch
>> > is tremendously faster than
>> > find [REPOSITORY_DIRECTORY_NAME] -exec touch {} \;
>> >
>> >
>> >
>> >
>> > Deng Ching-2 wrote:
>> >>
>> >> Hi Marc,
>> >>
>> >> Did you touch the contents of the repository? The last modified date
>> of
>> >> the
>> >> artifacts need to be more recent than the last run of the repository
>> >> scan.
>> >> You can execute *find [REPOSITORY_DIRECTORY_NAME] | xargs touch *at
>> the
>> >> base
>> >> dir where your repo is located.
>> >>
>> >> Thanks,
>> >> Deng
>> >>
>> >> On Wed, Jun 3, 2009 at 11:54 PM, Marc Lustig <ml...@marclustig.com>
>> wrote:
>> >>
>> >>>
>> >>> I forgot to add an excerpt from the log:
>> >>>
>> >>> 2009-06-03 17:42:11,554 [pool-2-thread-1] INFO
>> >>>
>> >>>
>> org.apache.maven.archiva.scheduled.executors.ArchivaRepositoryScanningTaskExecutor
>> >>> - Finished repository task:
>> >>> .\ Scan of internal \.__________________________________________
>> >>> Repository Dir : /opt/managed_repos/internal
>> >>> Repository Name : ....
>> >>> Repository Layout : default
>> >>> Known Consumers : (7 configured)
>> >>> auto-rename
>> >>> metadata-updater
>> >>> repository-purge
>> >>> auto-remove
>> >>> update-db-artifact
>> >>> create-missing-checksums
>> >>> index-content
>> >>> Invalid Consumers : <none>
>> >>> Duration : 1 Minute 13 Seconds 689 Milliseconds
>> >>> When Gathered : 6/3/09 5:42 PM
>> >>> Total File Count : 107263
>> >>> Avg Time Per File :
>> >>> ______________________________________________________________
>> >>> 2009-06-03 17:42:55,317 [pool-1-thread-1] INFO
>> >>>
>> >>>
>> org.apache.maven.archiva.scheduled.executors.ArchivaDatabaseUpdateTaskExecutor
>> >>> - Executing task from queue with job name:
>> database-job:user-requested
>> >>> 2009-06-03 17:42:55,319 [pool-1-thread-1] INFO
>> >>>
>> >>>
>> org.apache.maven.archiva.scheduled.executors.ArchivaDatabaseUpdateTaskExecutor
>> >>> - Task: Updating unprocessed artifacts
>> >>> 2009-06-03 17:42:55,318 [http-8080-6] INFO
>> >>> org.apache.maven.archiva.web.action.admin.SchedulerAction -
>> >>> [ActionMessage]
>> >>> Your request to update the database has been queued.
>> >>> 2009-06-03 17:42:55,591 [pool-1-thread-1] INFO
>> >>>
>> >>>
>> org.apache.maven.archiva.scheduled.executors.ArchivaDatabaseUpdateTaskExecutor
>> >>> - Task: Updating processed artifacts
>> >>> 2009-06-03 17:44:20,355 [pool-1-thread-1] INFO
>> >>>
>> >>>
>> org.apache.maven.archiva.scheduled.executors.ArchivaDatabaseUpdateTaskExecutor
>> >>> - Finished database task in 85035ms.
>> >>>
>> >>>
>> >>>
>> >>> Marc Lustig wrote:
>> >>> >
>> >>> > We upgraded our test-server to release 1.2.1, removed .index and
>> >>> .indexer
>> >>> > from all the repos, ran the Scan for the repos and then the Update
>> >>> > Database.
>> >>> >
>> >>> > As a result, before the upgrade we had 500 MB inside in all of the
>> >>> .index
>> >>> > directories.
>> >>> > After the upgrade and the repo-scan, the total size is less than 1
>> >>> MB.
>> >>> > Apparently the repo indexing-process does not work anymore. The
>> >>> > Search-function does not give any results.
>> >>> >
>> >>> > Any idea how to make the repos to get indexed again?
>> >>> >
>> >>>
>> >>> --
>> >>> View this message in context:
>> >>>
>> http://www.nabble.com/Archiva-1.2.1---repos-won%27t-get-index-tp23854158p23854203.html
>> >>> Sent from the archiva-users mailing list archive at Nabble.com.
>> >>>
>> >>>
>> >>
>> >>
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Archiva-1.2.1---repos-won%27t-get-index-tp23854158p24201482.html
>> Sent from the archiva-users mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Archiva-1.2.1---repos-won%27t-get-index-tp23854158p25474649.html
Sent from the archiva-users mailing list archive at Nabble.com.
Re: Archiva 1.2.1 - repos won't get index
Posted by Brett Porter <br...@apache.org>.
Perhaps you could use Luke to inspect the Lucene index and see whether
that helps identify the issue by looking at what data has been
successfully recorded?
On 17/09/2009, at 1:19 AM, Marc Lustig wrote:
>
> Hi,
> I gave this another try by doing the following steps.
>
> 1. deleting repo/.index and repo/.indexer
> 2. touching all files and dirs in repo recursively
> 3. running the repo scan
>
> When we search now for e.g. xmlbeans, we do get a few jar-artifacts
> that
> contain a package-name "xmlbeans". But we do not get the artifact
> whose name
> is xmlbeans.
>
> I have no idea what else to try to make the search-function to work
> again.
> User's here are pretty annoyed about it.
>
> Any suggestion what to do except filing a jira issue?
>
>
>
>
>
> Deng Ching-2 wrote:
>>
>> On Thu, Jun 25, 2009 at 7:26 PM, Marc Lustig <ml...@marclustig.com>
>> wrote:
>>
>>>
>>> On our side, the story is continuing like this.
>>> In the afternoon, I touch the whole repo and let the scan running
>>> and
>>> re-create the database.
>>> After that, it seems like all (physically) existing artifacts can be
>>> found
>>> on the Web-UI. Fine.
>>> The next morning again hundreds of artifacts are disappeared from
>>> the
>>> Web-UI
>>> (but still existing physically). It's so annoying...
>>> I checked the last couple of scans in the log and cannot find
>>> exceptions
>>> for
>>> that Repo concerned.
>>> (However I didn't check all the scans since yesterday afternoon.)
>>
>>
>>>
>>> Anyway, for some reason, some consumer is removing certain
>>> artifacts (or
>>> whole groupId's) from the index. I have no clue why.
>>>
>>> Any suggestion what to do to prevent the removal of artifacts from
>>> the
>>> index-database?
>>>
>>> If we disable the "update-db-artifact" consumer, what undesired
>>> side-effects
>>> may this have?
>>>
>>>
>> Your artifacts won't show up in the webapp browse as this consumer
>> adds
>> the
>> basic artifact info in the db and that's where the artifacts shown
>> in the
>> Browse page is gathered..
>>
>> You can try disabling these consumers in Database page as they are
>> the
>> ones
>> doing the cleanup work, but afaik they only clean up the db and
>> index for
>> artifacts that are no longer in the filesystem:
>> not-present-remove-db-artifact
>> not-present-remove-db-project
>> not-present-remove-indexed
>>
>> If the problem still persists, do you mind filing an issue for this
>> in
>> jira?
>> :)
>>
>> Thanks,
>> Deng
>>
>>
>>>
>>> Marc Lustig wrote:
>>>>
>>>> Indeed,
>>>> find [REPOSITORY_DIRECTORY_NAME] | xargs touch
>>>> is tremendously faster than
>>>> find [REPOSITORY_DIRECTORY_NAME] -exec touch {} \;
>>>>
>>>>
>>>>
>>>>
>>>> Deng Ching-2 wrote:
>>>>>
>>>>> Hi Marc,
>>>>>
>>>>> Did you touch the contents of the repository? The last modified
>>>>> date
>>> of
>>>>> the
>>>>> artifacts need to be more recent than the last run of the
>>>>> repository
>>>>> scan.
>>>>> You can execute *find [REPOSITORY_DIRECTORY_NAME] | xargs touch
>>>>> *at
>>> the
>>>>> base
>>>>> dir where your repo is located.
>>>>>
>>>>> Thanks,
>>>>> Deng
>>>>>
>>>>> On Wed, Jun 3, 2009 at 11:54 PM, Marc Lustig <ml...@marclustig.com>
>>> wrote:
>>>>>
>>>>>>
>>>>>> I forgot to add an excerpt from the log:
>>>>>>
>>>>>> 2009-06-03 17:42:11,554 [pool-2-thread-1] INFO
>>>>>>
>>>>>>
>>> org.apache.maven.archiva.scheduled.executors.ArchivaRepositoryScanningTaskExecutor
>>>>>> - Finished repository task:
>>>>>> .\ Scan of internal \.__________________________________________
>>>>>> Repository Dir : /opt/managed_repos/internal
>>>>>> Repository Name : ....
>>>>>> Repository Layout : default
>>>>>> Known Consumers : (7 configured)
>>>>>> auto-rename
>>>>>> metadata-updater
>>>>>> repository-purge
>>>>>> auto-remove
>>>>>> update-db-artifact
>>>>>> create-missing-checksums
>>>>>> index-content
>>>>>> Invalid Consumers : <none>
>>>>>> Duration : 1 Minute 13 Seconds 689 Milliseconds
>>>>>> When Gathered : 6/3/09 5:42 PM
>>>>>> Total File Count : 107263
>>>>>> Avg Time Per File :
>>>>>> ______________________________________________________________
>>>>>> 2009-06-03 17:42:55,317 [pool-1-thread-1] INFO
>>>>>>
>>>>>>
>>> org.apache.maven.archiva.scheduled.executors.ArchivaDatabaseUpdateTaskExecutor
>>>>>> - Executing task from queue with job name:
>>> database-job:user-requested
>>>>>> 2009-06-03 17:42:55,319 [pool-1-thread-1] INFO
>>>>>>
>>>>>>
>>> org.apache.maven.archiva.scheduled.executors.ArchivaDatabaseUpdateTaskExecutor
>>>>>> - Task: Updating unprocessed artifacts
>>>>>> 2009-06-03 17:42:55,318 [http-8080-6] INFO
>>>>>> org.apache.maven.archiva.web.action.admin.SchedulerAction -
>>>>>> [ActionMessage]
>>>>>> Your request to update the database has been queued.
>>>>>> 2009-06-03 17:42:55,591 [pool-1-thread-1] INFO
>>>>>>
>>>>>>
>>> org.apache.maven.archiva.scheduled.executors.ArchivaDatabaseUpdateTaskExecutor
>>>>>> - Task: Updating processed artifacts
>>>>>> 2009-06-03 17:44:20,355 [pool-1-thread-1] INFO
>>>>>>
>>>>>>
>>> org.apache.maven.archiva.scheduled.executors.ArchivaDatabaseUpdateTaskExecutor
>>>>>> - Finished database task in 85035ms.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Marc Lustig wrote:
>>>>>>>
>>>>>>> We upgraded our test-server to release 1.2.1, removed .index and
>>>>>> .indexer
>>>>>>> from all the repos, ran the Scan for the repos and then the
>>>>>>> Update
>>>>>>> Database.
>>>>>>>
>>>>>>> As a result, before the upgrade we had 500 MB inside in all of
>>>>>>> the
>>>>>> .index
>>>>>>> directories.
>>>>>>> After the upgrade and the repo-scan, the total size is less
>>>>>>> than 1
>>>>>> MB.
>>>>>>> Apparently the repo indexing-process does not work anymore. The
>>>>>>> Search-function does not give any results.
>>>>>>>
>>>>>>> Any idea how to make the repos to get indexed again?
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>>
>>> http://www.nabble.com/Archiva-1.2.1---repos-won%27t-get-index-tp23854158p23854203.html
>>>>>> Sent from the archiva-users mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Archiva-1.2.1---repos-won%27t-get-index-tp23854158p24201482.html
>>> Sent from the archiva-users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Archiva-1.2.1---repos-won%27t-get-index-tp23854158p25474649.html
> Sent from the archiva-users mailing list archive at Nabble.com.
>