You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@archiva.apache.org by Dan Armbrust <da...@gmail.com> on 2011/05/17 17:46:55 UTC

browse not showing all groups?

So, I uploaded some stuff with the web form, and most of the items I
uploaded behaved as I expect.

However, one of them refuses to appear in the Browse Repository
section of the GUI.  If I manually enter the path, it will at least
navigate to the folder (but it shows as empty).

If I go down the WebDav path, then everything appears as I would
expect.  Maven can also do downloads, as expected.

Anyone have any ideas how I ended up with this item magically hidden
from the GUI?

More importantly, how do I fix it?

Thanks

Re: browse not showing all groups?

Posted by Brett Porter <br...@apache.org>.
On 02/08/2011, at 10:37 AM, Dan Armbrust wrote:

>> 
>> Is there anything in the logs at the time of the upload, or the next
>> time it tries to index?  I vaguely recall an issue with a similar
>> version number...
>> 
>> -Wendy
>> 
> 
> I don't have access to the server in question, unfortunately.
> 
> I also don't have a copy running locally at the moment to test.  I
> hoped it would be an easy enough thing to test for a developer :)
> 
> I'm at a loss trying to understand how archiva deals with version
> numbers overall.
> 
> It seems to have arbitrary, undocumented behavior within the validator
> (at least on the gui upload).  I don't see any reason at all why
> archiva should have a say over the makeup and composition of a version
> number.  At most, it would seem that the version number would need to
> be valid as a part of a file name across all OS platforms.  But beyond
> that, why should it care?

In a Maven repository, we need to parse the path sometimes to be able to determine what the version and classifier are. That has been known to trip up on non-standard versions in the past and we've built up some test cases. It should be a simple bug to fix.

In trunk, we're moving to relying on that less - particularly in cases where it is not important (such as here where everything is a Maven 2 repository and there is a POM that describes the version for it). There's still some work to do there.

- Brett

--
Brett Porter
brett@apache.org
http://brettporter.wordpress.com/
http://au.linkedin.com/in/brettporter

Re: browse not showing all groups?

Posted by Wendy Smoak <ws...@gmail.com>.
On Tue, Aug 2, 2011 at 1:23 PM, Dan Armbrust
<da...@gmail.com> wrote:
> So, not only does archiva have a limitation that it really shouldn't
> have in the first place (I whole-heartedly agree with the previous
> poster - archiva should not be enforcing any rules on the makeup of a
> version numbers) - it also has a bug where it doesn't validate the
> input well enough to enforce its own limitations.

I'm not sure I'd want it validating and rejecting an upload because it
can't store the value in the database.  It would make the behavior
different for "mvn deploy" vs. the upload feature -- and it can still
store the artifact in the repository and serve it as a dependency.

I don't think it's enforcing rules this time, you've just run into an
edge case with a very long version number and an implementation detail
of an arbitrary length limit in a SQL database.  I much prefer
multi-value databases that don't have such issues. ;)

I don't know what's involved in increasing the length of that field,
whether you can just do it in the database or if you have to change
something in Archiva.

The error handling isn't the best, you really shouldn't have to dive
into the log files to figure out what's wrong.  (I thought there was a
repository health report, maybe it could show up there?)

To get past it for now... is all that _really_ version information, or
could some of it go in the artifactid?  At the very least, including
"-version" in the version number is redundant, and if you removed
that, you'd be under the 50 character limit.

-- 
Wendy

Re: browse not showing all groups?

Posted by Dan Armbrust <da...@gmail.com>.
Well, here is the cause of the "hidden" item:

2011-08-02 01:39:11,640 [pool-2-thread-1] ERROR
org.apache.maven.archiva.repository.scanner.functors.ConsumerProcessFileClosure
 - Consumer [update-db-artifact] had an error when processing file
[C:\Program Files
(x86)\archiva\apache-archiva-1.3.4\data\repositories\apelon-data\com\apelon\va\ihtsdo\wb\snomed\snomed-data\2010.07.31-build1-sct-en-vtmvmp-non-human-2-version\snomed-data-2010.07.31-build1-sct-en-vtmvmp-non-human-2-version.zip]:
Attempt to store value
"2010.07.31-build1-sct-en-vtmvmp-non-human-2-version" in column
"VERSION" that has maximum length of 50. Please correct your data!
javax.jdo.JDOFatalUserException: Attempt to store value
"2010.07.31-build1-sct-en-vtmvmp-non-human-2-version" in column
"VERSION" that has maximum length of 50. Please correct your data!
	at org.jpox.store.rdbms.mapping.CharRDBMSMapping.setString(CharRDBMSMapping.java:214)
	at org.jpox.store.mapping.SingleFieldMapping.setString(SingleFieldMapping.java:203)
	at org.jpox.store.rdbms.fieldmanager.ParameterSetter.storeStringField(ParameterSetter.java:122)
	at org.jpox.state.StateManagerImpl.providedStringField(StateManagerImpl.java:2757)
	at org.apache.maven.archiva.model.ArchivaArtifactModel.jdoProvideField(ArchivaArtifactModel.java)
	at org.apache.maven.archiva.model.ArchivaArtifactModel.jdoProvideFields(ArchivaArtifactModel.java)
	at org.jpox.state.StateManagerImpl.provideFields(StateManagerImpl.java:3115)
	at org.jpox.store.rdbms.request.InsertRequest.execute(InsertRequest.java:221)
	at org.jpox.store.rdbms.table.ClassTable.insert(ClassTable.java:2519)
	at org.jpox.store.StoreManager.insert(StoreManager.java:938)
	at org.jpox.state.StateManagerImpl.internalMakePersistent(StateManagerImpl.java:3667)
	at org.jpox.state.StateManagerImpl.makePersistent(StateManagerImpl.java:3646)
	at org.jpox.AbstractPersistenceManager.internalMakePersistent(AbstractPersistenceManager.java:1206)
	at org.jpox.AbstractPersistenceManager.makePersistent(AbstractPersistenceManager.java:1277)
	at org.apache.maven.archiva.database.jdo.JdoAccess.saveObject(JdoAccess.java:188)
	at org.apache.maven.archiva.database.jdo.JdoAccess.saveObject(JdoAccess.java:161)
	at org.apache.maven.archiva.database.jdo.JdoArtifactDAO.saveArtifact(JdoArtifactDAO.java:107)
	at org.apache.maven.archiva.consumers.database.ArtifactUpdateDatabaseConsumer.processFile(ArtifactUpdateDatabaseConsumer.java:211)
	at org.apache.maven.archiva.consumers.database.ArtifactUpdateDatabaseConsumer.processFile(ArtifactUpdateDatabaseConsumer.java:222)
	at org.apache.maven.archiva.repository.scanner.functors.ConsumerProcessFileClosure.execute(ConsumerProcessFileClosure.java:61)
	at org.apache.commons.collections.functors.IfClosure.execute(IfClosure.java:117)
	at org.apache.commons.collections.CollectionUtils.forAllDo(CollectionUtils.java:388)
	at org.apache.maven.archiva.repository.scanner.RepositoryScannerInstance.directoryWalkStep(RepositoryScannerInstance.java:161)
	at org.codehaus.plexus.util.DirectoryWalker.fireStep(DirectoryWalker.java:174)
	at org.codehaus.plexus.util.DirectoryWalker.scanDir(DirectoryWalker.java:392)
	at org.codehaus.plexus.util.DirectoryWalker.scanDir(DirectoryWalker.java:386)
	at org.codehaus.plexus.util.DirectoryWalker.scanDir(DirectoryWalker.java:386)
	at org.codehaus.plexus.util.DirectoryWalker.scanDir(DirectoryWalker.java:386)
	at org.codehaus.plexus.util.DirectoryWalker.scanDir(DirectoryWalker.java:386)
	at org.codehaus.plexus.util.DirectoryWalker.scanDir(DirectoryWalker.java:386)
	at org.codehaus.plexus.util.DirectoryWalker.scanDir(DirectoryWalker.java:386)
	at org.codehaus.plexus.util.DirectoryWalker.scanDir(DirectoryWalker.java:386)
	at org.codehaus.plexus.util.DirectoryWalker.scanDir(DirectoryWalker.java:386)
	at org.codehaus.plexus.util.DirectoryWalker.scan(DirectoryWalker.java:345)
	at org.apache.maven.archiva.repository.scanner.DefaultRepositoryScanner.scan(DefaultRepositoryScanner.java:125)
	at org.apache.maven.archiva.repository.scanner.DefaultRepositoryScanner.scan(DefaultRepositoryScanner.java:67)
	at org.apache.maven.archiva.scheduled.executors.ArchivaRepositoryScanningTaskExecutor.executeTask(ArchivaRepositoryScanningTaskExecutor.java:141)
	at org.codehaus.plexus.taskqueue.execution.ThreadedTaskQueueExecutor$ExecutorRunnable$1.run(ThreadedTaskQueueExecutor.java:116)
	at edu.emory.mathcs.backport.java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:442)
	at edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:176)
	at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:987)
	at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:528)
	at java.lang.Thread.run(Unknown Source)

So, not only does archiva have a limitation that it really shouldn't
have in the first place (I whole-heartedly agree with the previous
poster - archiva should not be enforcing any rules on the makeup of a
version numbers) - it also has a bug where it doesn't validate the
input well enough to enforce its own limitations.

Re: browse not showing all groups?

Posted by Dan Armbrust <da...@gmail.com>.
> This is the issue I remember:  https://jira.codehaus.org/browse/MRM-1172
>
> My argument was the same -- if Maven thinks it's okay as a version
> number, why would Archiva think differently?
>
> It sounds like you are able to upload the artifact though, so it's not
> quite the same issue.  My upload was failing and saying "invalid
> version".

I  have hit that a number of times too.  It's incredibly frustrating,
especially because the validation is done server side, after the
entire file upload (which takes over an hour in some cases for me).

Through trial and error, I've discovered that the first character must
be a number... and I know there were some other illegal things I
tried, but I don't recall them again now (which means I'll get bit by
them again in the future... sigh)

Re: browse not showing all groups?

Posted by Dan Armbrust <da...@gmail.com>.
> When you say it's hidden, where exactly are you looking?  Does it show
> up when you view the repository directly /archiva/repository/myrepo/
> and not when you browse with the html view
> /archiva/browse/groupid/artifact?


If I just go down the "Browse" path, it doesn't show the path to this
upload at all.  If I manually enter parts of the path on the URL, it
will go to the next level of the path.  But it won't show any files
there.

If I go to the WebDav path, however, everything is there.  Maven
clients have no issue accessing the "hidden" files... it just doesn't
show up when you browse the repository.

Re: browse not showing all groups?

Posted by Wendy Smoak <ws...@gmail.com>.
On Mon, Aug 1, 2011 at 8:37 PM, Dan Armbrust
<da...@gmail.com> wrote:
> I'm at a loss trying to understand how archiva deals with version
> numbers overall.
>
> It seems to have arbitrary, undocumented behavior within the validator
> (at least on the gui upload).  I don't see any reason at all why
> archiva should have a say over the makeup and composition of a version
> number.  At most, it would seem that the version number would need to
> be valid as a part of a file name across all OS platforms.  But beyond
> that, why should it care?

This is the issue I remember:  https://jira.codehaus.org/browse/MRM-1172

My argument was the same -- if Maven thinks it's okay as a version
number, why would Archiva think differently?

It sounds like you are able to upload the artifact though, so it's not
quite the same issue.  My upload was failing and saying "invalid
version".

When you say it's hidden, where exactly are you looking?  Does it show
up when you view the repository directly /archiva/repository/myrepo/
and not when you browse with the html view
/archiva/browse/groupid/artifact?

Sorry, I don't have the 1.3.x branch running to test it, I'm using trunk.

-- 
Wendy

Re: browse not showing all groups?

Posted by Dan Armbrust <da...@gmail.com>.
>
> Is there anything in the logs at the time of the upload, or the next
> time it tries to index?  I vaguely recall an issue with a similar
> version number...
>
> -Wendy
>

I don't have access to the server in question, unfortunately.

I also don't have a copy running locally at the moment to test.  I
hoped it would be an easy enough thing to test for a developer :)

I'm at a loss trying to understand how archiva deals with version
numbers overall.

It seems to have arbitrary, undocumented behavior within the validator
(at least on the gui upload).  I don't see any reason at all why
archiva should have a say over the makeup and composition of a version
number.  At most, it would seem that the version number would need to
be valid as a part of a file name across all OS platforms.  But beyond
that, why should it care?

Re: browse not showing all groups?

Posted by Wendy Smoak <ws...@gmail.com>.
On Mon, Aug 1, 2011 at 8:14 PM, Dan Armbrust
<da...@gmail.com> wrote:

> So, this nuisance of archiva not showing me certain things I've
> uploaded has come back again as well.
...
> I wonder if it is something with my version number.
>
> Could someone try uploading a junk zip file using a version number like this:
> 2010.07.31-build1-sct-en-vtmvmp-non-human-2-version
> It also failed to build the md5 sum files, etc, for this file, when
> the upload completed.

Is there anything in the logs at the time of the upload, or the next
time it tries to index?  I vaguely recall an issue with a similar
version number...

-Wendy

Re: browse not showing all groups?

Posted by Dan Armbrust <da...@gmail.com>.
So, this nuisance of archiva not showing me certain things I've
uploaded has come back again as well.

I thought this was a one time fluke, previously.  Since everything
else that I upload through the web form appears.

I finally just deleted the "hidden" one today, and re-uploaded it.

And it is hidden - again!

I wonder if it is something with my version number.

Could someone try uploading a junk zip file using a version number like this:

2010.07.31-build1-sct-en-vtmvmp-non-human-2-version

It also failed to build the md5 sum files, etc, for this file, when
the upload completed.

Re: browse not showing all groups?

Posted by Brett Porter <br...@apache.org>.
On 24/05/2011, at 11:06 PM, Dan Armbrust wrote:

> I've uploaded another unrelated file since then (to a different path),
> and it worked fine.  Just seems to be an issue with this specific
> path.

If that's the case, then there shouldn't be a configuration problem... but without access to the data it's also hard to diagnose. Maybe there is a stuck corrupt record in the database that needs to be removed. Certainly in trunk (what 1.4 will be based on), we've made improvements in this regard.

> 
> I'll keep poking at this nuisance once in a while.  It's just low on
> my priority list, since it more or less works at the moment.  And it
> doesn't help that I don't have direct access to the system.

Good luck - if you need help doing a more detailed investigation let us know.

- Brett

--
Brett Porter
brett@apache.org
http://brettporter.wordpress.com/
http://au.linkedin.com/in/brettporter

Re: browse not showing all groups?

Posted by Dan Armbrust <da...@gmail.com>.
On Wed, May 18, 2011 at 7:12 PM, Brett Porter <br...@apache.org> wrote:
>
> On 18/05/2011, at 4:43 AM, Dan Armbrust wrote:
>
>>>> Database -> Update Database Now
>>>> Repositories -> Scan Repository Now (with process all checked)
>>>> And just basic hard-refreshes through the browser, to make sure it
>>>> isn't a browser-cache issue.
>>>
>>> I believe those need to be done in reverse order.  (Though if you did
>>> it more than once, the second 'update database' should have caught
>>> what the scan turned up.)
>>>
>>
>>
>> Is there supposed to be some sort of feedback when I click those buttons?
>>
>> The submit seems to go through, but there is no feedback at all.
>
> It goes into a queue, which you can see on the queue page.


I don't see it there... but I think it happens so fast on this small
repository, that it is gone before I get there.

The last scanned time stamp seems reasonable.

>> Funny thing is, IIRC, it did work the first time I uploaded this
>> artifact.  But then, (after a very long time, due to the size of the
>> file) it decided to reject my upload, because it didn't like my
>> version number.  Sigh.  So many bugs in that version number
>> algorithm.... it should just be turned off if you ask me.
>>
>> So I did a delete, tried again.  Ran into another bug in the version
>> number verification... did a delete, tried again.  Finally found a
>> version number that it wouldn't reject.  But by then, the group had
>> vanished from the browser, and has never returned.
>>
>> Probably related to this sequence of events... I guess I could try to
>> reproduce it with a smaller file.  But first I'd need to set up a
>> local install that I can completely administer.  Too little time, at
>> the moment.
>
> That sounds quite strange. The instructions Wendy provided are complete - the only thing I can think of is that you are uploading some form of POM along with the artifact (or generating it)?
>
> Does the extension of the artifact in question in exist in the "artifacts" list on the "repository scanning page"?
>
> - Brett

Just uploaded a zip file.  And zip is still in the extensions list.  I
let it generate the pom - and when I browse it in the webdav view, it
all seems correct.  It made the pom, and it generated all the checksum
files, and the maven-metadata file.

I keep hoping it will magically fix itself... but so far, no luck.

I've uploaded another unrelated file since then (to a different path),
and it worked fine.  Just seems to be an issue with this specific
path.

I'll keep poking at this nuisance once in a while.  It's just low on
my priority list, since it more or less works at the moment.  And it
doesn't help that I don't have direct access to the system.

Re: browse not showing all groups?

Posted by Brett Porter <br...@apache.org>.
On 18/05/2011, at 4:43 AM, Dan Armbrust wrote:

>>> Database -> Update Database Now
>>> Repositories -> Scan Repository Now (with process all checked)
>>> And just basic hard-refreshes through the browser, to make sure it
>>> isn't a browser-cache issue.
>> 
>> I believe those need to be done in reverse order.  (Though if you did
>> it more than once, the second 'update database' should have caught
>> what the scan turned up.)
>> 
> 
> 
> Is there supposed to be some sort of feedback when I click those buttons?
> 
> The submit seems to go through, but there is no feedback at all.

It goes into a queue, which you can see on the queue page.

> 
> Funny thing is, IIRC, it did work the first time I uploaded this
> artifact.  But then, (after a very long time, due to the size of the
> file) it decided to reject my upload, because it didn't like my
> version number.  Sigh.  So many bugs in that version number
> algorithm.... it should just be turned off if you ask me.
> 
> So I did a delete, tried again.  Ran into another bug in the version
> number verification... did a delete, tried again.  Finally found a
> version number that it wouldn't reject.  But by then, the group had
> vanished from the browser, and has never returned.
> 
> Probably related to this sequence of events... I guess I could try to
> reproduce it with a smaller file.  But first I'd need to set up a
> local install that I can completely administer.  Too little time, at
> the moment.

That sounds quite strange. The instructions Wendy provided are complete - the only thing I can think of is that you are uploading some form of POM along with the artifact (or generating it)?

Does the extension of the artifact in question in exist in the "artifacts" list on the "repository scanning page"?

- Brett


--
Brett Porter
brett@apache.org
http://brettporter.wordpress.com/
http://au.linkedin.com/in/brettporter

Re: browse not showing all groups?

Posted by Dan Armbrust <da...@gmail.com>.
>> Database -> Update Database Now
>> Repositories -> Scan Repository Now (with process all checked)
>> And just basic hard-refreshes through the browser, to make sure it
>> isn't a browser-cache issue.
>
> I believe those need to be done in reverse order.  (Though if you did
> it more than once, the second 'update database' should have caught
> what the scan turned up.)
>


Is there supposed to be some sort of feedback when I click those buttons?

The submit seems to go through, but there is no feedback at all.

Funny thing is, IIRC, it did work the first time I uploaded this
artifact.  But then, (after a very long time, due to the size of the
file) it decided to reject my upload, because it didn't like my
version number.  Sigh.  So many bugs in that version number
algorithm.... it should just be turned off if you ask me.

So I did a delete, tried again.  Ran into another bug in the version
number verification... did a delete, tried again.  Finally found a
version number that it wouldn't reject.  But by then, the group had
vanished from the browser, and has never returned.

Probably related to this sequence of events... I guess I could try to
reproduce it with a smaller file.  But first I'd need to set up a
local install that I can completely administer.  Too little time, at
the moment.

Re: browse not showing all groups?

Posted by Wendy Smoak <ws...@gmail.com>.
On Tue, May 17, 2011 at 12:48 PM, Dan Armbrust
<da...@gmail.com> wrote:
> Database -> Update Database Now
> Repositories -> Scan Repository Now (with process all checked)
> And just basic hard-refreshes through the browser, to make sure it
> isn't a browser-cache issue.

I believe those need to be done in reverse order.  (Though if you did
it more than once, the second 'update database' should have caught
what the scan turned up.)

-- 
Wendy

Re: browse not showing all groups?

Posted by Dan Armbrust <da...@gmail.com>.
Well, I don't know what everything means.

But I've tried:

Database -> Update Database Now
Repositories -> Scan Repository Now (with process all checked)
And just basic hard-refreshes through the browser, to make sure it
isn't a browser-cache issue.

Tried clicking the "Flush" links on the system status page, but they
all appear to be unimplemented.

Unfortunately, I don't have direct access to the server itself, so I
can't go poking around the file system looking for anything out of the
ordinary.

Dan


On Tue, May 17, 2011 at 11:35 AM, Wendy Smoak <ws...@gmail.com> wrote:
> On Tue, May 17, 2011 at 12:23 PM, Dan Armbrust
> <da...@gmail.com> wrote:
>> Hmm.  I've clicked every refresh button I can find several times...
>> and even waited a couple of days to see if it was some sort of cache
>> issue.  Even tried adding / deleting something else within the same
>> group. It just never shows up.
>>
>> Version is 1.3.4.
>
> What exactly have you tried?  I don't know of a refresh button in Archiva.
>
> Let us know if scanning and updating fixes it, I suspect your upload
> happened during a scan and the timestamp is older than when Archiva
> thinks it last scanned.  Situations like that (as well as manually
> adding 'old' artifacts to the filesystem,) are why the checkbox to
> force it to scan *everything* was added.
>
> --
> Wendy
>

Re: browse not showing all groups?

Posted by Wendy Smoak <ws...@gmail.com>.
On Tue, May 17, 2011 at 12:23 PM, Dan Armbrust
<da...@gmail.com> wrote:
> Hmm.  I've clicked every refresh button I can find several times...
> and even waited a couple of days to see if it was some sort of cache
> issue.  Even tried adding / deleting something else within the same
> group. It just never shows up.
>
> Version is 1.3.4.

What exactly have you tried?  I don't know of a refresh button in Archiva.

Let us know if scanning and updating fixes it, I suspect your upload
happened during a scan and the timestamp is older than when Archiva
thinks it last scanned.  Situations like that (as well as manually
adding 'old' artifacts to the filesystem,) are why the checkbox to
force it to scan *everything* was added.

-- 
Wendy

Re: browse not showing all groups?

Posted by Dan Armbrust <da...@gmail.com>.
Hmm.  I've clicked every refresh button I can find several times...
and even waited a couple of days to see if it was some sort of cache
issue.  Even tried adding / deleting something else within the same
group. It just never shows up.

Version is 1.3.4.



On Tue, May 17, 2011 at 11:07 AM, Wendy Smoak <ws...@gmail.com> wrote:
> On Tue, May 17, 2011 at 11:46 AM, Dan Armbrust
> <da...@gmail.com> wrote:
>> So, I uploaded some stuff with the web form, and most of the items I
>> uploaded behaved as I expect.
>>
>> However, one of them refuses to appear in the Browse Repository
>> section of the GUI.  If I manually enter the path, it will at least
>> navigate to the folder (but it shows as empty).
>
> What version of Archiva?
>
> I think the /browse/ url works off the database contents rather than
> the actual filesystem.  Try scanning the repo and updating the
> database.   There is a checkbox to force it to scan *all* of the
> content rather than just what it thinks is 'new'.
>
> --
> Wendy
>

Re: browse not showing all groups?

Posted by Wendy Smoak <ws...@gmail.com>.
On Tue, May 17, 2011 at 11:46 AM, Dan Armbrust
<da...@gmail.com> wrote:
> So, I uploaded some stuff with the web form, and most of the items I
> uploaded behaved as I expect.
>
> However, one of them refuses to appear in the Browse Repository
> section of the GUI.  If I manually enter the path, it will at least
> navigate to the folder (but it shows as empty).

What version of Archiva?

I think the /browse/ url works off the database contents rather than
the actual filesystem.  Try scanning the repo and updating the
database.   There is a checkbox to force it to scan *all* of the
content rather than just what it thinks is 'new'.

-- 
Wendy