You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by Dogtail Ray <sp...@gmail.com> on 2015/09/21 22:20:06 UTC

How to modify Hadoop APIs used by Spark?

Hi all,

I find that Spark uses some Hadoop APIs such as InputFormat, InputSplit,
etc., and I want to modify these Hadoop APIs. Do you know how can I
integrate my modified Hadoop code into Spark? Great thanks!

Re: How to modify Hadoop APIs used by Spark?

Posted by Dogtail L <sp...@gmail.com>.

Oh, I want to modify existing Hadoop InputFormat.

On Mon, Sep 21, 2015 at 4:23 PM, Ted Yu <yu...@gmail.com> wrote:

> Can you clarify what you want to do:
> If you modify existing hadoop InputFormat, etc, it would be a matter of
> rebuilding hadoop and build Spark using the custom built hadoop as
> dependency.
>
> Do you introduce new InputFormat ?
>
> Cheers
>
> On Mon, Sep 21, 2015 at 1:20 PM, Dogtail Ray <sp...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I find that Spark uses some Hadoop APIs such as InputFormat, InputSplit,
>> etc., and I want to modify these Hadoop APIs. Do you know how can I
>> integrate my modified Hadoop code into Spark? Great thanks!
>>
>>
>

Re: How to modify Hadoop APIs used by Spark?

Posted by Ted Yu <yu...@gmail.com>.

Can you clarify what you want to do:
If you modify existing hadoop InputFormat, etc, it would be a matter of
rebuilding hadoop and build Spark using the custom built hadoop as
dependency.

Do you introduce new InputFormat ?

Cheers

On Mon, Sep 21, 2015 at 1:20 PM, Dogtail Ray <sp...@gmail.com> wrote:

> Hi all,
>
> I find that Spark uses some Hadoop APIs such as InputFormat, InputSplit,
> etc., and I want to modify these Hadoop APIs. Do you know how can I
> integrate my modified Hadoop code into Spark? Great thanks!
>
>