You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jean Georges Perrin <jg...@jgp.net> on 2016/10/04 21:35:54 UTC

Parsing XML

Spark 2.0.0
XML parser 0.4.0
Java

Hi,

I am trying to create a new column in my data frame, based on a value of a sub element. I have done that several time with JSON, but not very successful in XML.

(I know a world with less format would be easier :) )

Here is the code:
df.withColumn("FulfillmentOption1", df.col("//FulfillmentOption[1]/text()"));

And here is the error:
Exception in thread "main" org.apache.spark.sql.AnalysisException: Cannot resolve column name "//FulfillmentOption[1]/text()" among (x, xx, xxx, xxxx, a, b, FulfillmentOption, c, d, e, f, g);
    at org.apache.spark.sql.Dataset$$anonfun$resolve$1.apply(Dataset.scala:220)
    at org.apache.spark.sql.Dataset$$anonfun$resolve$1.apply(Dataset.scala:220)
    ...

The XPath is valid...

Thanks!

jg


---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Re: Parsing XML

Posted by Jean Georges Perrin <jg...@jgp.net>.
Yep... I was thinking about that... but it seems to work w JSON

jg


> On Oct 4, 2016, at 19:17, Peter Figliozzi <pe...@gmail.com> wrote:
> 
> It's pretty clear that df.col(xpath) is looking for a column named xpath in your df, not executing an xpath over an XML document as you wish.  Try constructing a UDF which applies your xpath query, and give that as the second argument to withColumn.
> 
>> On Tue, Oct 4, 2016 at 4:35 PM, Jean Georges Perrin <jg...@jgp.net> wrote:
>> Spark 2.0.0
>> XML parser 0.4.0
>> Java
>> 
>> Hi,
>> 
>> I am trying to create a new column in my data frame, based on a value of a sub element. I have done that several time with JSON, but not very successful in XML.
>> 
>> (I know a world with less format would be easier :) )
>> 
>> Here is the code:
>> df.withColumn("FulfillmentOption1", df.col("//FulfillmentOption[1]/text()"));
>> 
>> And here is the error:
>> Exception in thread "main" org.apache.spark.sql.AnalysisException: Cannot resolve column name "//FulfillmentOption[1]/text()" among (x, xx, xxx, xxxx, a, b, FulfillmentOption, c, d, e, f, g);
>>     at org.apache.spark.sql.Dataset$$anonfun$resolve$1.apply(Dataset.scala:220)
>>     at org.apache.spark.sql.Dataset$$anonfun$resolve$1.apply(Dataset.scala:220)
>>     ...
>> 
>> The XPath is valid...
>> 
>> Thanks!
>> 
>> jg
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>> 
> 

Re: Parsing XML

Posted by Peter Figliozzi <pe...@gmail.com>.
It's pretty clear that df.col(xpath) is looking for a column named xpath in
your df, not executing an xpath over an XML document as you wish.  Try
constructing a UDF which applies your xpath query, and give that as the
second argument to withColumn.

On Tue, Oct 4, 2016 at 4:35 PM, Jean Georges Perrin <jg...@jgp.net> wrote:

> Spark 2.0.0
> XML parser 0.4.0
> Java
>
> Hi,
>
> I am trying to create a new column in my data frame, based on a value of a
> sub element. I have done that several time with JSON, but not very
> successful in XML.
>
> (I know a world with less format would be easier :) )
>
> Here is the code:
> df.withColumn("FulfillmentOption1", df.col("//FulfillmentOption[1]
> /text()"));
>
> And here is the error:
> Exception in thread "main" org.apache.spark.sql.AnalysisException: Cannot
> resolve column name "//FulfillmentOption[1]/text()" among (x, xx, xxx,
> xxxx, a, b, FulfillmentOption, c, d, e, f, g);
>     at org.apache.spark.sql.Dataset$$anonfun$resolve$1.apply(
> Dataset.scala:220)
>     at org.apache.spark.sql.Dataset$$anonfun$resolve$1.apply(
> Dataset.scala:220)
>     ...
>
> The XPath is valid...
>
> Thanks!
>
> jg
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>