You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jakub Liska <li...@gmail.com> on 2016/03/09 13:38:37 UTC
DataFrame support for hadoop glob patterns
Hey,
is something like this possible?
sqlContext.read.json("/mnt/views-p/base/2016/01/*/*-xyz.json")
I switched to DataFrames because my source files changed from TSV to JSON
but now I'm not able to load the files as I did before. I get this error if
I try that :
https://github.com/apache/spark/pull/9142#issuecomment-194248531
Re: DataFrame support for hadoop glob patterns
Posted by Christophe Préaud <ch...@kelkoo.com>.
Hi,
Unless I've misunderstood what you want to achieve, you could use:
sqlContext.read.json(sc.textFile("/mnt/views-p/base/2016/01/*/*-xyz.json"))
Regards,
Christophe.
On 09/03/16 15:24, Ted Yu wrote:
Hadoop glob pattern doesn't support multi level wildcard.
Thanks
On Mar 9, 2016, at 6:15 AM, Koert Kuipers <<m...@tresata.com>> wrote:
if its based on HadoopFsRelation shouldn't it support it? HadoopFsRelation handles globs
On Wed, Mar 9, 2016 at 8:56 AM, Ted Yu <yu...@gmail.com>> wrote:
This is currently not supported.
On Mar 9, 2016, at 4:38 AM, Jakub Liska <<m...@gmail.com>> wrote:
Hey,
is something like this possible?
sqlContext.read.json("/mnt/views-p/base/2016/01/*/*-xyz.json")
I switched to DataFrames because my source files changed from TSV to JSON
but now I'm not able to load the files as I did before. I get this error if I try that :
https://github.com/apache/spark/pull/9142#issuecomment-194248531
________________________________
Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 158 Ter Rue du Temple 75003 Paris
425 093 069 RCS Paris
Ce message et les pièces jointes sont confidentiels et établis à l'attention exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce message, merci de le détruire et d'en avertir l'expéditeur.
Re: DataFrame support for hadoop glob patterns
Posted by Koert Kuipers <ko...@tresata.com>.
i tried with avro input, something like /data/txn_*/* and it works for me
On Wed, Mar 9, 2016 at 12:12 PM, Ted Yu <yu...@gmail.com> wrote:
> Koert:
> I meant org.apache.hadoop.mapred.FileInputFormat doesn't support multi
> level wildcard.
>
> Cheers
>
> On Wed, Mar 9, 2016 at 8:22 AM, Koert Kuipers <ko...@tresata.com> wrote:
>
>> i use multi level wildcard with hadoop fs -ls, which is the exact same
>> glob function call
>>
>> On Wed, Mar 9, 2016 at 9:24 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> Hadoop glob pattern doesn't support multi level wildcard.
>>>
>>> Thanks
>>>
>>> On Mar 9, 2016, at 6:15 AM, Koert Kuipers <ko...@tresata.com> wrote:
>>>
>>> if its based on HadoopFsRelation shouldn't it support it?
>>> HadoopFsRelation handles globs
>>>
>>> On Wed, Mar 9, 2016 at 8:56 AM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> This is currently not supported.
>>>>
>>>> On Mar 9, 2016, at 4:38 AM, Jakub Liska <li...@gmail.com> wrote:
>>>>
>>>> Hey,
>>>>
>>>> is something like this possible?
>>>>
>>>> sqlContext.read.json("/mnt/views-p/base/2016/01/*/*-xyz.json")
>>>>
>>>> I switched to DataFrames because my source files changed from TSV to
>>>> JSON
>>>> but now I'm not able to load the files as I did before. I get this
>>>> error if I try that :
>>>>
>>>> https://github.com/apache/spark/pull/9142#issuecomment-194248531
>>>>
>>>>
>>>
>>
>
Re: DataFrame support for hadoop glob patterns
Posted by Ted Yu <yu...@gmail.com>.
Koert:
I meant org.apache.hadoop.mapred.FileInputFormat doesn't support multi
level wildcard.
Cheers
On Wed, Mar 9, 2016 at 8:22 AM, Koert Kuipers <ko...@tresata.com> wrote:
> i use multi level wildcard with hadoop fs -ls, which is the exact same
> glob function call
>
> On Wed, Mar 9, 2016 at 9:24 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> Hadoop glob pattern doesn't support multi level wildcard.
>>
>> Thanks
>>
>> On Mar 9, 2016, at 6:15 AM, Koert Kuipers <ko...@tresata.com> wrote:
>>
>> if its based on HadoopFsRelation shouldn't it support it?
>> HadoopFsRelation handles globs
>>
>> On Wed, Mar 9, 2016 at 8:56 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> This is currently not supported.
>>>
>>> On Mar 9, 2016, at 4:38 AM, Jakub Liska <li...@gmail.com> wrote:
>>>
>>> Hey,
>>>
>>> is something like this possible?
>>>
>>> sqlContext.read.json("/mnt/views-p/base/2016/01/*/*-xyz.json")
>>>
>>> I switched to DataFrames because my source files changed from TSV to JSON
>>> but now I'm not able to load the files as I did before. I get this error
>>> if I try that :
>>>
>>> https://github.com/apache/spark/pull/9142#issuecomment-194248531
>>>
>>>
>>
>
Re: DataFrame support for hadoop glob patterns
Posted by Koert Kuipers <ko...@tresata.com>.
i use multi level wildcard with hadoop fs -ls, which is the exact same glob
function call
On Wed, Mar 9, 2016 at 9:24 AM, Ted Yu <yu...@gmail.com> wrote:
> Hadoop glob pattern doesn't support multi level wildcard.
>
> Thanks
>
> On Mar 9, 2016, at 6:15 AM, Koert Kuipers <ko...@tresata.com> wrote:
>
> if its based on HadoopFsRelation shouldn't it support it? HadoopFsRelation
> handles globs
>
> On Wed, Mar 9, 2016 at 8:56 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> This is currently not supported.
>>
>> On Mar 9, 2016, at 4:38 AM, Jakub Liska <li...@gmail.com> wrote:
>>
>> Hey,
>>
>> is something like this possible?
>>
>> sqlContext.read.json("/mnt/views-p/base/2016/01/*/*-xyz.json")
>>
>> I switched to DataFrames because my source files changed from TSV to JSON
>> but now I'm not able to load the files as I did before. I get this error
>> if I try that :
>>
>> https://github.com/apache/spark/pull/9142#issuecomment-194248531
>>
>>
>
Re: DataFrame support for hadoop glob patterns
Posted by Ted Yu <yu...@gmail.com>.
Hadoop glob pattern doesn't support multi level wildcard.
Thanks
> On Mar 9, 2016, at 6:15 AM, Koert Kuipers <ko...@tresata.com> wrote:
>
> if its based on HadoopFsRelation shouldn't it support it? HadoopFsRelation handles globs
>
>> On Wed, Mar 9, 2016 at 8:56 AM, Ted Yu <yu...@gmail.com> wrote:
>> This is currently not supported.
>>
>>> On Mar 9, 2016, at 4:38 AM, Jakub Liska <li...@gmail.com> wrote:
>>>
>>> Hey,
>>>
>>> is something like this possible?
>>>
>>> sqlContext.read.json("/mnt/views-p/base/2016/01/*/*-xyz.json")
>>>
>>> I switched to DataFrames because my source files changed from TSV to JSON
>>> but now I'm not able to load the files as I did before. I get this error if I try that :
>>>
>>> https://github.com/apache/spark/pull/9142#issuecomment-194248531
>
Re: DataFrame support for hadoop glob patterns
Posted by Koert Kuipers <ko...@tresata.com>.
if its based on HadoopFsRelation shouldn't it support it? HadoopFsRelation
handles globs
On Wed, Mar 9, 2016 at 8:56 AM, Ted Yu <yu...@gmail.com> wrote:
> This is currently not supported.
>
> On Mar 9, 2016, at 4:38 AM, Jakub Liska <li...@gmail.com> wrote:
>
> Hey,
>
> is something like this possible?
>
> sqlContext.read.json("/mnt/views-p/base/2016/01/*/*-xyz.json")
>
> I switched to DataFrames because my source files changed from TSV to JSON
> but now I'm not able to load the files as I did before. I get this error
> if I try that :
>
> https://github.com/apache/spark/pull/9142#issuecomment-194248531
>
>
Re: DataFrame support for hadoop glob patterns
Posted by Ted Yu <yu...@gmail.com>.
This is currently not supported.
> On Mar 9, 2016, at 4:38 AM, Jakub Liska <li...@gmail.com> wrote:
>
> Hey,
>
> is something like this possible?
>
> sqlContext.read.json("/mnt/views-p/base/2016/01/*/*-xyz.json")
>
> I switched to DataFrames because my source files changed from TSV to JSON
> but now I'm not able to load the files as I did before. I get this error if I try that :
>
> https://github.com/apache/spark/pull/9142#issuecomment-194248531