You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by Ryan Blue <bl...@cloudera.com> on 2015/03/06 20:08:10 UTC
Broken build
It looks like the hadoop-2 tests are breaking in master from this commit:
https://github.com/apache/incubator-parquet-mr/commit/5851e6da
The problem is that mergeFooters doesn't find any footers. In Hadoop-1,
the data files are in the output directory, but on hadoop-2 the output
file is here:
target/test/ParquetTBaseScheme/names-parquet-out/_temporary/0/task_local567920263_0001_m_000000/part-00000-m-00000.parquet
Anyone want to volunteer to fix it? I'm happy to revert the change,
since I'm not sure what the right fix for the bug is.
Also, this is a good time to remind everyone to please check Travis CI
or run the hadoop-2 profile locally before merging. Thanks!
rb
--
Ryan Blue
Software Engineer
Cloudera, Inc.
Re: Broken build
Posted by Ryan Blue <bl...@cloudera.com>.
On 03/06/2015 03:53 PM, Tianshuo Deng wrote:
> sending out a revert https://github.com/apache/incubator-parquet-mr/pull/139
Thanks Tianshuo! Go ahead and push reverts like that when you've used
`git revert`. I don't think that specifically needs reviewed, unless the
revert can't happen cleanly because of newer changes, which isn't the
case here.
rb
--
Ryan Blue
Software Engineer
Cloudera, Inc.
Re: Broken build
Posted by Tianshuo Deng <td...@twitter.com.INVALID>.
sending out a revert https://github.com/apache/incubator-parquet-mr/pull/139
On Fri, Mar 6, 2015 at 1:39 PM, Tianshuo Deng <td...@twitter.com> wrote:
> fix: https://github.com/apache/incubator-parquet-mr/pull/138
>
> On Fri, Mar 6, 2015 at 11:26 AM, Tianshuo Deng <td...@twitter.com> wrote:
>
>> checking, will try to fix it
>>
>> On Fri, Mar 6, 2015 at 11:12 AM, Julien Le Dem <ju...@twitter.com>
>> wrote:
>>
>>> My guess is that the committer should call super.
>>> public class MapredParquetOutputCommitter extends FileOutputCommitter {
>>> @Override
>>> public void commitJob(JobContext jobContext) throws IOException {
>>> super.commitJob(jobContext);
>>> ...
>>> }
>>> }
>>> Tianshuo, do you prefer trying to fix it or reverting?
>>> We should not keep a failing build in master
>>>
>>>
>>> On Fri, Mar 6, 2015 at 11:08 AM, Ryan Blue <bl...@cloudera.com> wrote:
>>>
>>>> It looks like the hadoop-2 tests are breaking in master from this
>>>> commit:
>>>>
>>>> https://github.com/apache/incubator-parquet-mr/commit/5851e6da
>>>>
>>>> The problem is that mergeFooters doesn't find any footers. In Hadoop-1,
>>>> the data files are in the output directory, but on hadoop-2 the output file
>>>> is here:
>>>>
>>>>
>>>> target/test/ParquetTBaseScheme/names-parquet-out/_temporary/0/task_
>>>> local567920263_0001_m_000000/part-00000-m-00000.parquet
>>>>
>>>> Anyone want to volunteer to fix it? I'm happy to revert the change,
>>>> since I'm not sure what the right fix for the bug is.
>>>>
>>>> Also, this is a good time to remind everyone to please check Travis CI
>>>> or run the hadoop-2 profile locally before merging. Thanks!
>>>>
>>>> rb
>>>>
>>>> --
>>>> Ryan Blue
>>>> Software Engineer
>>>> Cloudera, Inc.
>>>>
>>>
>>>
>>
>
Re: Broken build
Posted by Tianshuo Deng <td...@twitter.com.INVALID>.
fix: https://github.com/apache/incubator-parquet-mr/pull/138
On Fri, Mar 6, 2015 at 11:26 AM, Tianshuo Deng <td...@twitter.com> wrote:
> checking, will try to fix it
>
> On Fri, Mar 6, 2015 at 11:12 AM, Julien Le Dem <ju...@twitter.com> wrote:
>
>> My guess is that the committer should call super.
>> public class MapredParquetOutputCommitter extends FileOutputCommitter {
>> @Override
>> public void commitJob(JobContext jobContext) throws IOException {
>> super.commitJob(jobContext);
>> ...
>> }
>> }
>> Tianshuo, do you prefer trying to fix it or reverting?
>> We should not keep a failing build in master
>>
>>
>> On Fri, Mar 6, 2015 at 11:08 AM, Ryan Blue <bl...@cloudera.com> wrote:
>>
>>> It looks like the hadoop-2 tests are breaking in master from this commit:
>>>
>>> https://github.com/apache/incubator-parquet-mr/commit/5851e6da
>>>
>>> The problem is that mergeFooters doesn't find any footers. In Hadoop-1,
>>> the data files are in the output directory, but on hadoop-2 the output file
>>> is here:
>>>
>>>
>>> target/test/ParquetTBaseScheme/names-parquet-out/_temporary/0/task_
>>> local567920263_0001_m_000000/part-00000-m-00000.parquet
>>>
>>> Anyone want to volunteer to fix it? I'm happy to revert the change,
>>> since I'm not sure what the right fix for the bug is.
>>>
>>> Also, this is a good time to remind everyone to please check Travis CI
>>> or run the hadoop-2 profile locally before merging. Thanks!
>>>
>>> rb
>>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Cloudera, Inc.
>>>
>>
>>
>
Re: Broken build
Posted by Tianshuo Deng <td...@twitter.com.INVALID>.
checking, will try to fix it
On Fri, Mar 6, 2015 at 11:12 AM, Julien Le Dem <ju...@twitter.com> wrote:
> My guess is that the committer should call super.
> public class MapredParquetOutputCommitter extends FileOutputCommitter {
> @Override
> public void commitJob(JobContext jobContext) throws IOException {
> super.commitJob(jobContext);
> ...
> }
> }
> Tianshuo, do you prefer trying to fix it or reverting?
> We should not keep a failing build in master
>
>
> On Fri, Mar 6, 2015 at 11:08 AM, Ryan Blue <bl...@cloudera.com> wrote:
>
>> It looks like the hadoop-2 tests are breaking in master from this commit:
>>
>> https://github.com/apache/incubator-parquet-mr/commit/5851e6da
>>
>> The problem is that mergeFooters doesn't find any footers. In Hadoop-1,
>> the data files are in the output directory, but on hadoop-2 the output file
>> is here:
>>
>>
>> target/test/ParquetTBaseScheme/names-parquet-out/_temporary/0/task_
>> local567920263_0001_m_000000/part-00000-m-00000.parquet
>>
>> Anyone want to volunteer to fix it? I'm happy to revert the change, since
>> I'm not sure what the right fix for the bug is.
>>
>> Also, this is a good time to remind everyone to please check Travis CI or
>> run the hadoop-2 profile locally before merging. Thanks!
>>
>> rb
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Cloudera, Inc.
>>
>
>
Re: Broken build
Posted by Julien Le Dem <ju...@twitter.com.INVALID>.
My guess is that the committer should call super.
public class MapredParquetOutputCommitter extends FileOutputCommitter {
@Override
public void commitJob(JobContext jobContext) throws IOException {
super.commitJob(jobContext);
...
}
}
Tianshuo, do you prefer trying to fix it or reverting?
We should not keep a failing build in master
On Fri, Mar 6, 2015 at 11:08 AM, Ryan Blue <bl...@cloudera.com> wrote:
> It looks like the hadoop-2 tests are breaking in master from this commit:
>
> https://github.com/apache/incubator-parquet-mr/commit/5851e6da
>
> The problem is that mergeFooters doesn't find any footers. In Hadoop-1,
> the data files are in the output directory, but on hadoop-2 the output file
> is here:
>
>
> target/test/ParquetTBaseScheme/names-parquet-out/_temporary/0/task_
> local567920263_0001_m_000000/part-00000-m-00000.parquet
>
> Anyone want to volunteer to fix it? I'm happy to revert the change, since
> I'm not sure what the right fix for the bug is.
>
> Also, this is a good time to remind everyone to please check Travis CI or
> run the hadoop-2 profile locally before merging. Thanks!
>
> rb
>
> --
> Ryan Blue
> Software Engineer
> Cloudera, Inc.
>