You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by Sean Owen <sr...@gmail.com> on 2010/10/14 15:02:29 UTC
Release tomorrow?
If there are no major issues, we'll start a release tomorrow evening. Any
major issues? I see some ongoing work here.
There's no hard deadline; we keep working if we need to. Just as long as
we're converging on something quickly.
Re: Release tomorrow?
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
What about the Building mahout issue? It seems several users are
experiencing build failures, including me.
On 10/14/10 6:02 AM, Sean Owen wrote:
> If there are no major issues, we'll start a release tomorrow evening. Any
> major issues? I see some ongoing work here.
> There's no hard deadline; we keep working if we need to. Just as long as
> we're converging on something quickly.
>
Re: Release tomorrow?
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
Found it! CHD3 adds a _SUCCESS file to the vectors directory and the
RandomSeedGenerator was trying to process it as a vector file. I'm
testing a patch which fixes k-means on Reuters.
On 10/16/10 8:32 AM, Jeff Eastman wrote:
> I'm able to run both Canopy and LDA on CHD3 after the first parts of
> build-reuters.sh (thru seq2sparse) have completed. The k-means job
> fails consistently in the RandomSeedGenerator. I'm investigating what
> may be different about its file handling compared to the other jobs.
>
> On 10/14/10 9:36 PM, Jeff Eastman wrote:
>> Well, in this case k-means fails the same way even after I've
>> verified the input file so its a hard failure. And the job runs just
>> fine stand-alone on the box, and in both modes on my Mac, so its got
>> to be something about the Cloudera deployment. Sure would be nice to
>> have 0.4 run that example on CDH3.
>>
>> On 10/14/10 8:53 PM, Ted Dunning wrote:
>>> There is often a small delay before files appear in HDFS after they are
>>> created. This has buggered many a work-flow.
>>>
>>> On Thu, Oct 14, 2010 at 8:40 PM, Jeff
>>> Eastman<jd...@windwardsolutions.com>wrote:
>>>
>>>> On 10/14/10 7:47 PM, Jeff Eastman wrote:
>>>>
>>>>> The recent commit to the POM fixed my build problem on my clean
>>>>> RedHat
>>>>> box. Currently, build-reuters.sh is failing to run the k-means
>>>>> step on
>>>>> Hadoop on that box and it looks like it is the same problem we've
>>>>> been
>>>>> seeing with others running the Cloudera CDH3: hadoop is running
>>>>> under a
>>>>> different user and the local file references don't resolve
>>>>> correctly when
>>>>> the job is run under mine. I haven't yet figured out the best way
>>>>> to fix
>>>>> this or why the other build-reuters job steps don't have this
>>>>> problem (they
>>>>> all use ./examples... file paths too).
>>>>>
>>>> It looks like the RandomSeedGenerator.buildRandom() is somehow
>>>> seeing an
>>>> empty input directory when it really has an 11.6 mb part file in
>>>> it. The
>>>> EOFException occurs when executing: SequenceFile.Reader reader = new
>>>> SequenceFile.Reader(fs, fileStatus.getPath(), conf); on line 84.
>>>> There are
>>>> hdfs and mapred PIDs associated with the hadoop daemons, but why
>>>> would that
>>>> matter? The files in hdfs are all under /users/dev/examples... and
>>>> my jobs
>>>> are running as dev so I don't get why this is happening.
>>>>
>>
>
Re: Release tomorrow?
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
I'm able to run both Canopy and LDA on CHD3 after the first parts of
build-reuters.sh (thru seq2sparse) have completed. The k-means job fails
consistently in the RandomSeedGenerator. I'm investigating what may be
different about its file handling compared to the other jobs.
On 10/14/10 9:36 PM, Jeff Eastman wrote:
> Well, in this case k-means fails the same way even after I've
> verified the input file so its a hard failure. And the job runs just
> fine stand-alone on the box, and in both modes on my Mac, so its got
> to be something about the Cloudera deployment. Sure would be nice to
> have 0.4 run that example on CDH3.
>
> On 10/14/10 8:53 PM, Ted Dunning wrote:
>> There is often a small delay before files appear in HDFS after they are
>> created. This has buggered many a work-flow.
>>
>> On Thu, Oct 14, 2010 at 8:40 PM, Jeff
>> Eastman<jd...@windwardsolutions.com>wrote:
>>
>>> On 10/14/10 7:47 PM, Jeff Eastman wrote:
>>>
>>>> The recent commit to the POM fixed my build problem on my clean
>>>> RedHat
>>>> box. Currently, build-reuters.sh is failing to run the k-means step on
>>>> Hadoop on that box and it looks like it is the same problem we've been
>>>> seeing with others running the Cloudera CDH3: hadoop is running
>>>> under a
>>>> different user and the local file references don't resolve
>>>> correctly when
>>>> the job is run under mine. I haven't yet figured out the best way
>>>> to fix
>>>> this or why the other build-reuters job steps don't have this
>>>> problem (they
>>>> all use ./examples... file paths too).
>>>>
>>> It looks like the RandomSeedGenerator.buildRandom() is somehow
>>> seeing an
>>> empty input directory when it really has an 11.6 mb part file in it.
>>> The
>>> EOFException occurs when executing: SequenceFile.Reader reader = new
>>> SequenceFile.Reader(fs, fileStatus.getPath(), conf); on line 84.
>>> There are
>>> hdfs and mapred PIDs associated with the hadoop daemons, but why
>>> would that
>>> matter? The files in hdfs are all under /users/dev/examples... and
>>> my jobs
>>> are running as dev so I don't get why this is happening.
>>>
>
Re: Release tomorrow?
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
Well, in this case k-means fails the same way even after I've verified
the input file so its a hard failure. And the job runs just fine
stand-alone on the box, and in both modes on my Mac, so its got to be
something about the Cloudera deployment. Sure would be nice to have 0.4
run that example on CDH3.
On 10/14/10 8:53 PM, Ted Dunning wrote:
> There is often a small delay before files appear in HDFS after they are
> created. This has buggered many a work-flow.
>
> On Thu, Oct 14, 2010 at 8:40 PM, Jeff Eastman<jd...@windwardsolutions.com>wrote:
>
>> On 10/14/10 7:47 PM, Jeff Eastman wrote:
>>
>>> The recent commit to the POM fixed my build problem on my clean RedHat
>>> box. Currently, build-reuters.sh is failing to run the k-means step on
>>> Hadoop on that box and it looks like it is the same problem we've been
>>> seeing with others running the Cloudera CDH3: hadoop is running under a
>>> different user and the local file references don't resolve correctly when
>>> the job is run under mine. I haven't yet figured out the best way to fix
>>> this or why the other build-reuters job steps don't have this problem (they
>>> all use ./examples... file paths too).
>>>
>> It looks like the RandomSeedGenerator.buildRandom() is somehow seeing an
>> empty input directory when it really has an 11.6 mb part file in it. The
>> EOFException occurs when executing: SequenceFile.Reader reader = new
>> SequenceFile.Reader(fs, fileStatus.getPath(), conf); on line 84. There are
>> hdfs and mapred PIDs associated with the hadoop daemons, but why would that
>> matter? The files in hdfs are all under /users/dev/examples... and my jobs
>> are running as dev so I don't get why this is happening.
>>
Re: Release tomorrow?
Posted by Ted Dunning <te...@gmail.com>.
There is often a small delay before files appear in HDFS after they are
created. This has buggered many a work-flow.
On Thu, Oct 14, 2010 at 8:40 PM, Jeff Eastman <jd...@windwardsolutions.com>wrote:
> On 10/14/10 7:47 PM, Jeff Eastman wrote:
>
>> The recent commit to the POM fixed my build problem on my clean RedHat
>> box. Currently, build-reuters.sh is failing to run the k-means step on
>> Hadoop on that box and it looks like it is the same problem we've been
>> seeing with others running the Cloudera CDH3: hadoop is running under a
>> different user and the local file references don't resolve correctly when
>> the job is run under mine. I haven't yet figured out the best way to fix
>> this or why the other build-reuters job steps don't have this problem (they
>> all use ./examples... file paths too).
>>
> It looks like the RandomSeedGenerator.buildRandom() is somehow seeing an
> empty input directory when it really has an 11.6 mb part file in it. The
> EOFException occurs when executing: SequenceFile.Reader reader = new
> SequenceFile.Reader(fs, fileStatus.getPath(), conf); on line 84. There are
> hdfs and mapred PIDs associated with the hadoop daemons, but why would that
> matter? The files in hdfs are all under /users/dev/examples... and my jobs
> are running as dev so I don't get why this is happening.
>
Re: Release tomorrow?
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
On 10/14/10 7:47 PM, Jeff Eastman wrote:
> The recent commit to the POM fixed my build problem on my clean
> RedHat box. Currently, build-reuters.sh is failing to run the k-means
> step on Hadoop on that box and it looks like it is the same problem
> we've been seeing with others running the Cloudera CDH3: hadoop is
> running under a different user and the local file references don't
> resolve correctly when the job is run under mine. I haven't yet
> figured out the best way to fix this or why the other build-reuters
> job steps don't have this problem (they all use ./examples... file
> paths too).
It looks like the RandomSeedGenerator.buildRandom() is somehow seeing an
empty input directory when it really has an 11.6 mb part file in it. The
EOFException occurs when executing: SequenceFile.Reader reader = new
SequenceFile.Reader(fs, fileStatus.getPath(), conf); on line 84. There
are hdfs and mapred PIDs associated with the hadoop daemons, but why
would that matter? The files in hdfs are all under
/users/dev/examples... and my jobs are running as dev so I don't get why
this is happening.
Re: Release tomorrow?
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
The recent commit to the POM fixed my build problem on my clean RedHat
box. Currently, build-reuters.sh is failing to run the k-means step on
Hadoop on that box and it looks like it is the same problem we've been
seeing with others running the Cloudera CDH3: hadoop is running under a
different user and the local file references don't resolve correctly
when the job is run under mine. I haven't yet figured out the best way
to fix this or why the other build-reuters job steps don't have this
problem (they all use ./examples... file paths too).
But yeah, we're pretty close.
On 10/14/10 11:11 AM, Grant Ingersoll wrote:
> Never mind, noise on my part. I had some patched code. Trying again.
>
> -Grant
>
> On Oct 14, 2010, at 2:09 PM, Grant Ingersoll wrote:
>
>> I'm getting test failures from a clean install call. It's weird, though, I don't see an obvious place where it printed out that it failed. Still digging.
>>
>>
>> On Oct 14, 2010, at 10:43 AM, Jeff Eastman wrote:
>>
>>> My build issue was not with Eclipse but with Maven but I'm suspecting a permissions issue and will investigate today and report later.
>>>
>>> On 10/14/10 7:13 AM, Sean Owen wrote:
>>>> Exactly, yes, go ahead and tag and build and vote on the product.
>>>>
>>>> We definitely can't proceed if it doesn't build but seems to be some issue
>>>> with Eclipse rather than Maven?
>>>>
>>>> On Thu, Oct 14, 2010 at 3:00 PM, Grant Ingersoll<gs...@apache.org>wrote:
>>>>
>>>>> Just to clarify, you mean start the release vote for a given set of
>>>>> artifacts, right?
>>>>>
>>>>> On Oct 14, 2010, at 9:02 AM, Sean Owen wrote:
>>>>>
>>>>>> If there are no major issues, we'll start a release tomorrow evening. Any
>>>>>> major issues? I see some ongoing work here.
>>>>>> There's no hard deadline; we keep working if we need to. Just as long as
>>>>>> we're converging on something quickly.
>>>>> --------------------------
>>>>> Grant Ingersoll
>>>>> http://www.lucidimagination.com
>>>>>
>>>>>
>
Re: Release tomorrow?
Posted by Grant Ingersoll <gs...@apache.org>.
Never mind, noise on my part. I had some patched code. Trying again.
-Grant
On Oct 14, 2010, at 2:09 PM, Grant Ingersoll wrote:
> I'm getting test failures from a clean install call. It's weird, though, I don't see an obvious place where it printed out that it failed. Still digging.
>
>
> On Oct 14, 2010, at 10:43 AM, Jeff Eastman wrote:
>
>> My build issue was not with Eclipse but with Maven but I'm suspecting a permissions issue and will investigate today and report later.
>>
>> On 10/14/10 7:13 AM, Sean Owen wrote:
>>> Exactly, yes, go ahead and tag and build and vote on the product.
>>>
>>> We definitely can't proceed if it doesn't build but seems to be some issue
>>> with Eclipse rather than Maven?
>>>
>>> On Thu, Oct 14, 2010 at 3:00 PM, Grant Ingersoll<gs...@apache.org>wrote:
>>>
>>>> Just to clarify, you mean start the release vote for a given set of
>>>> artifacts, right?
>>>>
>>>> On Oct 14, 2010, at 9:02 AM, Sean Owen wrote:
>>>>
>>>>> If there are no major issues, we'll start a release tomorrow evening. Any
>>>>> major issues? I see some ongoing work here.
>>>>> There's no hard deadline; we keep working if we need to. Just as long as
>>>>> we're converging on something quickly.
>>>> --------------------------
>>>> Grant Ingersoll
>>>> http://www.lucidimagination.com
>>>>
>>>>
>>
Re: Release tomorrow?
Posted by Grant Ingersoll <gs...@apache.org>.
I'm getting test failures from a clean install call. It's weird, though, I don't see an obvious place where it printed out that it failed. Still digging.
On Oct 14, 2010, at 10:43 AM, Jeff Eastman wrote:
> My build issue was not with Eclipse but with Maven but I'm suspecting a permissions issue and will investigate today and report later.
>
> On 10/14/10 7:13 AM, Sean Owen wrote:
>> Exactly, yes, go ahead and tag and build and vote on the product.
>>
>> We definitely can't proceed if it doesn't build but seems to be some issue
>> with Eclipse rather than Maven?
>>
>> On Thu, Oct 14, 2010 at 3:00 PM, Grant Ingersoll<gs...@apache.org>wrote:
>>
>>> Just to clarify, you mean start the release vote for a given set of
>>> artifacts, right?
>>>
>>> On Oct 14, 2010, at 9:02 AM, Sean Owen wrote:
>>>
>>>> If there are no major issues, we'll start a release tomorrow evening. Any
>>>> major issues? I see some ongoing work here.
>>>> There's no hard deadline; we keep working if we need to. Just as long as
>>>> we're converging on something quickly.
>>> --------------------------
>>> Grant Ingersoll
>>> http://www.lucidimagination.com
>>>
>>>
>
--------------------------
Grant Ingersoll
http://www.lucidimagination.com
Re: Release tomorrow?
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
My build issue was not with Eclipse but with Maven but I'm suspecting
a permissions issue and will investigate today and report later.
On 10/14/10 7:13 AM, Sean Owen wrote:
> Exactly, yes, go ahead and tag and build and vote on the product.
>
> We definitely can't proceed if it doesn't build but seems to be some issue
> with Eclipse rather than Maven?
>
> On Thu, Oct 14, 2010 at 3:00 PM, Grant Ingersoll<gs...@apache.org>wrote:
>
>> Just to clarify, you mean start the release vote for a given set of
>> artifacts, right?
>>
>> On Oct 14, 2010, at 9:02 AM, Sean Owen wrote:
>>
>>> If there are no major issues, we'll start a release tomorrow evening. Any
>>> major issues? I see some ongoing work here.
>>> There's no hard deadline; we keep working if we need to. Just as long as
>>> we're converging on something quickly.
>> --------------------------
>> Grant Ingersoll
>> http://www.lucidimagination.com
>>
>>
Re: Release tomorrow?
Posted by Sean Owen <sr...@gmail.com>.
Exactly, yes, go ahead and tag and build and vote on the product.
We definitely can't proceed if it doesn't build but seems to be some issue
with Eclipse rather than Maven?
On Thu, Oct 14, 2010 at 3:00 PM, Grant Ingersoll <gs...@apache.org>wrote:
> Just to clarify, you mean start the release vote for a given set of
> artifacts, right?
>
> On Oct 14, 2010, at 9:02 AM, Sean Owen wrote:
>
> > If there are no major issues, we'll start a release tomorrow evening. Any
> > major issues? I see some ongoing work here.
> > There's no hard deadline; we keep working if we need to. Just as long as
> > we're converging on something quickly.
>
> --------------------------
> Grant Ingersoll
> http://www.lucidimagination.com
>
>
Re: Release tomorrow?
Posted by Grant Ingersoll <gs...@apache.org>.
Just to clarify, you mean start the release vote for a given set of artifacts, right?
On Oct 14, 2010, at 9:02 AM, Sean Owen wrote:
> If there are no major issues, we'll start a release tomorrow evening. Any
> major issues? I see some ongoing work here.
> There's no hard deadline; we keep working if we need to. Just as long as
> we're converging on something quickly.
--------------------------
Grant Ingersoll
http://www.lucidimagination.com