You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by anu <an...@gmail.com> on 2015/02/24 11:09:16 UTC
Facing error while extending scala class with Product interface to
overcome limit of 22 fields in spark-shell
My issue is posted here on stack-overflow. What am I doing wrong here?
http://stackoverflow.com/questions/28689186/facing-error-while-extending-scala-class-with-product-interface-to-overcome-limi
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Facing-error-while-extending-scala-class-with-Product-interface-to-overcome-limit-of-22-fields-in-spl-tp21787.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Facing error while extending scala class with Product interface
to overcome limit of 22 fields in spark-shell
Posted by Patrick Varilly <pa...@dataminded.be>.
By the way, the limitation of case classes to 22 parameters was removed in
<https://issues.scala-lang.org/browse/SI-7296> Scala 2.11
<https://issues.scala-lang.org/browse/SI-7098> (there's some technical
rough edge <https://github.com/scala/scala/pull/2305> past 22 that you most
likely will never run into, but past 255, you run into underlying
limitations of the JVM <https://issues.scala-lang.org/browse/SI-7324>).
Best,
Patrick
On Thu, Feb 26, 2015 at 11:58 AM, anamika gupta <an...@gmail.com>
wrote:
> Hi Patrick
>
> Thanks a ton for your in-depth answer. The compilation error is now
> resolved.
>
> Thanks a lot again !!
>
> On Thu, Feb 26, 2015 at 2:40 PM, Patrick Varilly <
> patrick.varilly@dataminded.be> wrote:
>
>> Hi, Akhil,
>>
>> In your definition of sdp_d
>> <http://stackoverflow.com/questions/28689186/facing-error-while-extending-scala-class-with-product-interface-to-overcome-limi>,
>> all your fields are of type Option[X]. In Scala, a value of type Option[X]
>> can hold one of two things:
>>
>> 1. None
>> 2. Some(x), where x is of type X
>>
>> So to fix your immediate problem, wrap all your parameters to the sdp_d
>> constructor in Some(...), as follows:
>>
>> new sdp_d(Some(r(0).trim.toInt), Some(r(1).trim.toInt),
>> Some(r(2).trim), ...
>>
>> Your earlier question of why writing sdp_d(...) for a case class works
>> but you need to write new sdp_d(...) for an explicit class, there's a
>> simple answer. When you create a case class X in scala, Scala also makes a
>> companion object X behind the scenes with an apply method that calls new
>> (see below). Scala's rules will call this apply method automatically. So,
>> when you write "X(...)", you're really calling "X.apply(...)" which in turn
>> calls "new X(...)". (This is the same trick behind writing things like
>> List(1,2,3)) If you don't use a case class, you'd have to make the
>> companion object yourself explicitly.
>>
>> For reference, this statement:
>>
>> case class X(a: A, b: B)
>>
>> is conceptually equivalent to
>>
>> class X(val a: A, val b: B) extends ... {
>>
>> override def toString: String = // Auto-generated
>> override def hashCode: Int = // Auto-generated
>> override def equals(that: Any): Boolean = // Auto-generated
>>
>> ... more convenience methods ...
>> }
>>
>> object X {
>> def apply(a: A, b: B) = new X(a, b)
>> ... more convenience methods ...
>> }
>>
>> If you want to peek under the hood, try compiling a simple X.scala file
>> with the line "case class X(a: Int, b: Double)", then taking apart the
>> generated X.class and X$.class (e.g., "javap X.class").
>>
>> More info here
>> <http://docs.scala-lang.org/tutorials/tour/case-classes.html>, here
>> <http://www.scala-lang.org/docu/files/ScalaReference.pdf> and in Programming
>> in Scala <http://www.artima.com/shop/programming_in_scala_2ed> ch 15.
>>
>> Hope that helps!
>>
>> Best,
>>
>> Patrick
>>
>> On Thu, Feb 26, 2015 at 6:37 AM, anamika gupta <an...@gmail.com>
>> wrote:
>>
>>> I am now getting the following error. I cross-checked my types and
>>> corrected three of them i.e. r26-->String, r27-->Timestamp,
>>> r28-->Timestamp. This error still persists.
>>>
>>> scala>
>>> sc.textFile("/home/cdhuser/Desktop/Sdp_d.csv").map(_.split(",")).map { r =>
>>> | val upto_time = sdf.parse(r(23).trim);
>>> | calendar.setTime(upto_time);
>>> | val r23 = new java.sql.Timestamp(upto_time.getTime)
>>> | val insert_time = sdf.parse(r(27).trim)
>>> | calendar.setTime(insert_time)
>>> | val r27 = new java.sql.Timestamp(insert_time.getTime)
>>> | val last_upd_time = sdf.parse(r(28).trim)
>>> | calendar.setTime(last_upd_time)
>>> | val r28 = new java.sql.Timestamp(last_upd_time.getTime)
>>> | new sdp_d(r(0).trim.toInt, r(1).trim.toInt, r(2).trim,
>>> r(3).trim.toInt, r(4).trim.toInt, r(5).trim, r(6).trim.toInt, r(7).trim,
>>> r(8).trim.toDouble, r(9).trim.toDouble, r(10).trim, r(11).trim, r(12).trim,
>>> r(13).trim, r(14).trim, r(15).trim, r(16).trim, r(17).trim, r(18).trim,
>>> r(19).trim, r(20).trim, r(21).trim.toInt, r(22).trim, r23, r(24).trim,
>>> r(25).trim, r(26).trim, r27, r28)
>>> | }.registerAsTable("sdp_d")
>>>
>>> <console>:26: error: type mismatch;
>>> found : Int
>>> required: Option[Int]
>>> new sdp_d(r(0).trim.toInt, r(1).trim.toInt, r(2).trim,
>>> r(3).trim.toInt, r(4).trim.toInt, r(5).trim, r(6).trim.toInt, r(7).trim,
>>> r(8).trim.toDouble, r(9).trim.toDouble, r(10).trim, r(11).trim, r(12).trim,
>>> r(13).trim, r(14).trim, r(15).trim, r(16).trim, r(17).trim, r(18).trim,
>>> r(19).trim, r(20).trim, r(21).trim.toInt, r(22).trim, r23, r(24).trim,
>>> r(25).trim, r(26).trim, r27, r28)
>>>
>>> On Wed, Feb 25, 2015 at 2:32 PM, Akhil Das <ak...@sigmoidanalytics.com>
>>> wrote:
>>>
>>>> It says sdp_d not found, since it is a class you need to instantiate it
>>>> once. like:
>>>>
>>>> sc.textFile("derby.log").map(_.split(",")).map( r => {
>>>> val upto_time = sdf.parse(r(23).trim);
>>>> calendar.setTime(upto_time);
>>>> val r23 = new java.sql.Timestamp(upto_time.getTime);
>>>>
>>>> val insert_time = sdf.parse(r(26).trim);
>>>> calendar.setTime(insert_time);
>>>> val r26 = new java.sql.Timestamp(insert_time.getTime);
>>>>
>>>> val last_upd_time = sdf.parse(r(27).trim);
>>>> calendar.setTime(last_upd_time);
>>>> val r27 = new java.sql.Timestamp(last_upd_time.getTime);
>>>>
>>>> *new* *sdp_d(r(0).trim.toInt, r(1).trim.toInt, r(2).trim,
>>>> r(3).trim.toInt, r(4).trim.toInt, r(5).trim, r(6).trim.toInt, r(7).trim,
>>>> r(8).trim.toDouble, r(9).trim.toDouble, r(10).trim, r(11).trim, r(12).trim,
>>>> r(13).trim, r(14).trim, r(15).trim, r(16).trim, r(17).trim, r(18).trim,
>>>> r(19).trim, r(20).trim, r(21).trim.toInt, r(22).trim, r23, r(24).trim,
>>>> r(25).trim, r26, r27, r(28).trim)*
>>>> }).registerAsTable("sdp")
>>>>
>>>> Thanks
>>>> Best Regards
>>>>
>>>> On Wed, Feb 25, 2015 at 2:14 PM, anamika gupta <
>>>> anamika.guopta@gmail.com> wrote:
>>>>
>>>>> The link has proved helpful. I have been able to load data, register
>>>>> it as a table and perform simple queries. Thanks Akhil !!
>>>>>
>>>>> Though, I still look forward to knowing where I was going wrong with
>>>>> my previous technique of extending the Product Interface to overcome case
>>>>> class's limit of 22 fields.
>>>>>
>>>>> On Wed, Feb 25, 2015 at 9:45 AM, anamika gupta <
>>>>> anamika.guopta@gmail.com> wrote:
>>>>>
>>>>>> Hi Akhil
>>>>>>
>>>>>> I guess it skipped my attention. I would definitely give it a try.
>>>>>>
>>>>>> While I would still like to know what is the issue with the way I
>>>>>> have created schema?
>>>>>>
>>>>>> On Tue, Feb 24, 2015 at 4:35 PM, Akhil Das <
>>>>>> akhil@sigmoidanalytics.com> wrote:
>>>>>>
>>>>>>> Did you happen to have a look at
>>>>>>> https://spark.apache.org/docs/latest/sql-programming-guide.html#programmatically-specifying-the-schema
>>>>>>>
>>>>>>> Thanks
>>>>>>> Best Regards
>>>>>>>
>>>>>>> On Tue, Feb 24, 2015 at 3:39 PM, anu <an...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> My issue is posted here on stack-overflow. What am I doing wrong
>>>>>>>> here?
>>>>>>>>
>>>>>>>>
>>>>>>>> http://stackoverflow.com/questions/28689186/facing-error-while-extending-scala-class-with-product-interface-to-overcome-limi
>>>>>>>>
>>>>>>>> ------------------------------
>>>>>>>> View this message in context: Facing error while extending scala
>>>>>>>> class with Product interface to overcome limit of 22 fields in spark-shell
>>>>>>>> <http://apache-spark-user-list.1001560.n3.nabble.com/Facing-error-while-extending-scala-class-with-Product-interface-to-overcome-limit-of-22-fields-in-spl-tp21787.html>
>>>>>>>> Sent from the Apache Spark User List mailing list archive
>>>>>>>> <http://apache-spark-user-list.1001560.n3.nabble.com/> at
>>>>>>>> Nabble.com.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Facing error while extending scala class with Product interface
to overcome limit of 22 fields in spark-shell
Posted by anamika gupta <an...@gmail.com>.
Hi Patrick
Thanks a ton for your in-depth answer. The compilation error is now
resolved.
Thanks a lot again !!
On Thu, Feb 26, 2015 at 2:40 PM, Patrick Varilly <
patrick.varilly@dataminded.be> wrote:
> Hi, Akhil,
>
> In your definition of sdp_d
> <http://stackoverflow.com/questions/28689186/facing-error-while-extending-scala-class-with-product-interface-to-overcome-limi>,
> all your fields are of type Option[X]. In Scala, a value of type Option[X]
> can hold one of two things:
>
> 1. None
> 2. Some(x), where x is of type X
>
> So to fix your immediate problem, wrap all your parameters to the sdp_d
> constructor in Some(...), as follows:
>
> new sdp_d(Some(r(0).trim.toInt), Some(r(1).trim.toInt),
> Some(r(2).trim), ...
>
> Your earlier question of why writing sdp_d(...) for a case class works but
> you need to write new sdp_d(...) for an explicit class, there's a simple
> answer. When you create a case class X in scala, Scala also makes a
> companion object X behind the scenes with an apply method that calls new
> (see below). Scala's rules will call this apply method automatically. So,
> when you write "X(...)", you're really calling "X.apply(...)" which in turn
> calls "new X(...)". (This is the same trick behind writing things like
> List(1,2,3)) If you don't use a case class, you'd have to make the
> companion object yourself explicitly.
>
> For reference, this statement:
>
> case class X(a: A, b: B)
>
> is conceptually equivalent to
>
> class X(val a: A, val b: B) extends ... {
>
> override def toString: String = // Auto-generated
> override def hashCode: Int = // Auto-generated
> override def equals(that: Any): Boolean = // Auto-generated
>
> ... more convenience methods ...
> }
>
> object X {
> def apply(a: A, b: B) = new X(a, b)
> ... more convenience methods ...
> }
>
> If you want to peek under the hood, try compiling a simple X.scala file
> with the line "case class X(a: Int, b: Double)", then taking apart the
> generated X.class and X$.class (e.g., "javap X.class").
>
> More info here
> <http://docs.scala-lang.org/tutorials/tour/case-classes.html>, here
> <http://www.scala-lang.org/docu/files/ScalaReference.pdf> and in Programming
> in Scala <http://www.artima.com/shop/programming_in_scala_2ed> ch 15.
>
> Hope that helps!
>
> Best,
>
> Patrick
>
> On Thu, Feb 26, 2015 at 6:37 AM, anamika gupta <an...@gmail.com>
> wrote:
>
>> I am now getting the following error. I cross-checked my types and
>> corrected three of them i.e. r26-->String, r27-->Timestamp,
>> r28-->Timestamp. This error still persists.
>>
>> scala>
>> sc.textFile("/home/cdhuser/Desktop/Sdp_d.csv").map(_.split(",")).map { r =>
>> | val upto_time = sdf.parse(r(23).trim);
>> | calendar.setTime(upto_time);
>> | val r23 = new java.sql.Timestamp(upto_time.getTime)
>> | val insert_time = sdf.parse(r(27).trim)
>> | calendar.setTime(insert_time)
>> | val r27 = new java.sql.Timestamp(insert_time.getTime)
>> | val last_upd_time = sdf.parse(r(28).trim)
>> | calendar.setTime(last_upd_time)
>> | val r28 = new java.sql.Timestamp(last_upd_time.getTime)
>> | new sdp_d(r(0).trim.toInt, r(1).trim.toInt, r(2).trim,
>> r(3).trim.toInt, r(4).trim.toInt, r(5).trim, r(6).trim.toInt, r(7).trim,
>> r(8).trim.toDouble, r(9).trim.toDouble, r(10).trim, r(11).trim, r(12).trim,
>> r(13).trim, r(14).trim, r(15).trim, r(16).trim, r(17).trim, r(18).trim,
>> r(19).trim, r(20).trim, r(21).trim.toInt, r(22).trim, r23, r(24).trim,
>> r(25).trim, r(26).trim, r27, r28)
>> | }.registerAsTable("sdp_d")
>>
>> <console>:26: error: type mismatch;
>> found : Int
>> required: Option[Int]
>> new sdp_d(r(0).trim.toInt, r(1).trim.toInt, r(2).trim,
>> r(3).trim.toInt, r(4).trim.toInt, r(5).trim, r(6).trim.toInt, r(7).trim,
>> r(8).trim.toDouble, r(9).trim.toDouble, r(10).trim, r(11).trim, r(12).trim,
>> r(13).trim, r(14).trim, r(15).trim, r(16).trim, r(17).trim, r(18).trim,
>> r(19).trim, r(20).trim, r(21).trim.toInt, r(22).trim, r23, r(24).trim,
>> r(25).trim, r(26).trim, r27, r28)
>>
>> On Wed, Feb 25, 2015 at 2:32 PM, Akhil Das <ak...@sigmoidanalytics.com>
>> wrote:
>>
>>> It says sdp_d not found, since it is a class you need to instantiate it
>>> once. like:
>>>
>>> sc.textFile("derby.log").map(_.split(",")).map( r => {
>>> val upto_time = sdf.parse(r(23).trim);
>>> calendar.setTime(upto_time);
>>> val r23 = new java.sql.Timestamp(upto_time.getTime);
>>>
>>> val insert_time = sdf.parse(r(26).trim);
>>> calendar.setTime(insert_time);
>>> val r26 = new java.sql.Timestamp(insert_time.getTime);
>>>
>>> val last_upd_time = sdf.parse(r(27).trim);
>>> calendar.setTime(last_upd_time);
>>> val r27 = new java.sql.Timestamp(last_upd_time.getTime);
>>>
>>> *new* *sdp_d(r(0).trim.toInt, r(1).trim.toInt, r(2).trim,
>>> r(3).trim.toInt, r(4).trim.toInt, r(5).trim, r(6).trim.toInt, r(7).trim,
>>> r(8).trim.toDouble, r(9).trim.toDouble, r(10).trim, r(11).trim, r(12).trim,
>>> r(13).trim, r(14).trim, r(15).trim, r(16).trim, r(17).trim, r(18).trim,
>>> r(19).trim, r(20).trim, r(21).trim.toInt, r(22).trim, r23, r(24).trim,
>>> r(25).trim, r26, r27, r(28).trim)*
>>> }).registerAsTable("sdp")
>>>
>>> Thanks
>>> Best Regards
>>>
>>> On Wed, Feb 25, 2015 at 2:14 PM, anamika gupta <anamika.guopta@gmail.com
>>> > wrote:
>>>
>>>> The link has proved helpful. I have been able to load data, register it
>>>> as a table and perform simple queries. Thanks Akhil !!
>>>>
>>>> Though, I still look forward to knowing where I was going wrong with my
>>>> previous technique of extending the Product Interface to overcome case
>>>> class's limit of 22 fields.
>>>>
>>>> On Wed, Feb 25, 2015 at 9:45 AM, anamika gupta <
>>>> anamika.guopta@gmail.com> wrote:
>>>>
>>>>> Hi Akhil
>>>>>
>>>>> I guess it skipped my attention. I would definitely give it a try.
>>>>>
>>>>> While I would still like to know what is the issue with the way I have
>>>>> created schema?
>>>>>
>>>>> On Tue, Feb 24, 2015 at 4:35 PM, Akhil Das <akhil@sigmoidanalytics.com
>>>>> > wrote:
>>>>>
>>>>>> Did you happen to have a look at
>>>>>> https://spark.apache.org/docs/latest/sql-programming-guide.html#programmatically-specifying-the-schema
>>>>>>
>>>>>> Thanks
>>>>>> Best Regards
>>>>>>
>>>>>> On Tue, Feb 24, 2015 at 3:39 PM, anu <an...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> My issue is posted here on stack-overflow. What am I doing wrong
>>>>>>> here?
>>>>>>>
>>>>>>>
>>>>>>> http://stackoverflow.com/questions/28689186/facing-error-while-extending-scala-class-with-product-interface-to-overcome-limi
>>>>>>>
>>>>>>> ------------------------------
>>>>>>> View this message in context: Facing error while extending scala
>>>>>>> class with Product interface to overcome limit of 22 fields in spark-shell
>>>>>>> <http://apache-spark-user-list.1001560.n3.nabble.com/Facing-error-while-extending-scala-class-with-Product-interface-to-overcome-limit-of-22-fields-in-spl-tp21787.html>
>>>>>>> Sent from the Apache Spark User List mailing list archive
>>>>>>> <http://apache-spark-user-list.1001560.n3.nabble.com/> at
>>>>>>> Nabble.com.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Facing error while extending scala class with Product interface
to overcome limit of 22 fields in spark-shell
Posted by Patrick Varilly <pa...@dataminded.be>.
Hi, Akhil,
In your definition of sdp_d
<http://stackoverflow.com/questions/28689186/facing-error-while-extending-scala-class-with-product-interface-to-overcome-limi>,
all your fields are of type Option[X]. In Scala, a value of type Option[X]
can hold one of two things:
1. None
2. Some(x), where x is of type X
So to fix your immediate problem, wrap all your parameters to the sdp_d
constructor in Some(...), as follows:
new sdp_d(Some(r(0).trim.toInt), Some(r(1).trim.toInt), Some(r(2).trim),
...
Your earlier question of why writing sdp_d(...) for a case class works but
you need to write new sdp_d(...) for an explicit class, there's a simple
answer. When you create a case class X in scala, Scala also makes a
companion object X behind the scenes with an apply method that calls new
(see below). Scala's rules will call this apply method automatically. So,
when you write "X(...)", you're really calling "X.apply(...)" which in turn
calls "new X(...)". (This is the same trick behind writing things like
List(1,2,3)) If you don't use a case class, you'd have to make the
companion object yourself explicitly.
For reference, this statement:
case class X(a: A, b: B)
is conceptually equivalent to
class X(val a: A, val b: B) extends ... {
override def toString: String = // Auto-generated
override def hashCode: Int = // Auto-generated
override def equals(that: Any): Boolean = // Auto-generated
... more convenience methods ...
}
object X {
def apply(a: A, b: B) = new X(a, b)
... more convenience methods ...
}
If you want to peek under the hood, try compiling a simple X.scala file
with the line "case class X(a: Int, b: Double)", then taking apart the
generated X.class and X$.class (e.g., "javap X.class").
More info here <http://docs.scala-lang.org/tutorials/tour/case-classes.html>,
here <http://www.scala-lang.org/docu/files/ScalaReference.pdf> and in
Programming
in Scala <http://www.artima.com/shop/programming_in_scala_2ed> ch 15.
Hope that helps!
Best,
Patrick
On Thu, Feb 26, 2015 at 6:37 AM, anamika gupta <an...@gmail.com>
wrote:
> I am now getting the following error. I cross-checked my types and
> corrected three of them i.e. r26-->String, r27-->Timestamp,
> r28-->Timestamp. This error still persists.
>
> scala>
> sc.textFile("/home/cdhuser/Desktop/Sdp_d.csv").map(_.split(",")).map { r =>
> | val upto_time = sdf.parse(r(23).trim);
> | calendar.setTime(upto_time);
> | val r23 = new java.sql.Timestamp(upto_time.getTime)
> | val insert_time = sdf.parse(r(27).trim)
> | calendar.setTime(insert_time)
> | val r27 = new java.sql.Timestamp(insert_time.getTime)
> | val last_upd_time = sdf.parse(r(28).trim)
> | calendar.setTime(last_upd_time)
> | val r28 = new java.sql.Timestamp(last_upd_time.getTime)
> | new sdp_d(r(0).trim.toInt, r(1).trim.toInt, r(2).trim,
> r(3).trim.toInt, r(4).trim.toInt, r(5).trim, r(6).trim.toInt, r(7).trim,
> r(8).trim.toDouble, r(9).trim.toDouble, r(10).trim, r(11).trim, r(12).trim,
> r(13).trim, r(14).trim, r(15).trim, r(16).trim, r(17).trim, r(18).trim,
> r(19).trim, r(20).trim, r(21).trim.toInt, r(22).trim, r23, r(24).trim,
> r(25).trim, r(26).trim, r27, r28)
> | }.registerAsTable("sdp_d")
>
> <console>:26: error: type mismatch;
> found : Int
> required: Option[Int]
> new sdp_d(r(0).trim.toInt, r(1).trim.toInt, r(2).trim,
> r(3).trim.toInt, r(4).trim.toInt, r(5).trim, r(6).trim.toInt, r(7).trim,
> r(8).trim.toDouble, r(9).trim.toDouble, r(10).trim, r(11).trim, r(12).trim,
> r(13).trim, r(14).trim, r(15).trim, r(16).trim, r(17).trim, r(18).trim,
> r(19).trim, r(20).trim, r(21).trim.toInt, r(22).trim, r23, r(24).trim,
> r(25).trim, r(26).trim, r27, r28)
>
> On Wed, Feb 25, 2015 at 2:32 PM, Akhil Das <ak...@sigmoidanalytics.com>
> wrote:
>
>> It says sdp_d not found, since it is a class you need to instantiate it
>> once. like:
>>
>> sc.textFile("derby.log").map(_.split(",")).map( r => {
>> val upto_time = sdf.parse(r(23).trim);
>> calendar.setTime(upto_time);
>> val r23 = new java.sql.Timestamp(upto_time.getTime);
>>
>> val insert_time = sdf.parse(r(26).trim);
>> calendar.setTime(insert_time);
>> val r26 = new java.sql.Timestamp(insert_time.getTime);
>>
>> val last_upd_time = sdf.parse(r(27).trim);
>> calendar.setTime(last_upd_time);
>> val r27 = new java.sql.Timestamp(last_upd_time.getTime);
>>
>> *new* *sdp_d(r(0).trim.toInt, r(1).trim.toInt, r(2).trim,
>> r(3).trim.toInt, r(4).trim.toInt, r(5).trim, r(6).trim.toInt, r(7).trim,
>> r(8).trim.toDouble, r(9).trim.toDouble, r(10).trim, r(11).trim, r(12).trim,
>> r(13).trim, r(14).trim, r(15).trim, r(16).trim, r(17).trim, r(18).trim,
>> r(19).trim, r(20).trim, r(21).trim.toInt, r(22).trim, r23, r(24).trim,
>> r(25).trim, r26, r27, r(28).trim)*
>> }).registerAsTable("sdp")
>>
>> Thanks
>> Best Regards
>>
>> On Wed, Feb 25, 2015 at 2:14 PM, anamika gupta <an...@gmail.com>
>> wrote:
>>
>>> The link has proved helpful. I have been able to load data, register it
>>> as a table and perform simple queries. Thanks Akhil !!
>>>
>>> Though, I still look forward to knowing where I was going wrong with my
>>> previous technique of extending the Product Interface to overcome case
>>> class's limit of 22 fields.
>>>
>>> On Wed, Feb 25, 2015 at 9:45 AM, anamika gupta <anamika.guopta@gmail.com
>>> > wrote:
>>>
>>>> Hi Akhil
>>>>
>>>> I guess it skipped my attention. I would definitely give it a try.
>>>>
>>>> While I would still like to know what is the issue with the way I have
>>>> created schema?
>>>>
>>>> On Tue, Feb 24, 2015 at 4:35 PM, Akhil Das <ak...@sigmoidanalytics.com>
>>>> wrote:
>>>>
>>>>> Did you happen to have a look at
>>>>> https://spark.apache.org/docs/latest/sql-programming-guide.html#programmatically-specifying-the-schema
>>>>>
>>>>> Thanks
>>>>> Best Regards
>>>>>
>>>>> On Tue, Feb 24, 2015 at 3:39 PM, anu <an...@gmail.com> wrote:
>>>>>
>>>>>> My issue is posted here on stack-overflow. What am I doing wrong here?
>>>>>>
>>>>>>
>>>>>> http://stackoverflow.com/questions/28689186/facing-error-while-extending-scala-class-with-product-interface-to-overcome-limi
>>>>>>
>>>>>> ------------------------------
>>>>>> View this message in context: Facing error while extending scala
>>>>>> class with Product interface to overcome limit of 22 fields in spark-shell
>>>>>> <http://apache-spark-user-list.1001560.n3.nabble.com/Facing-error-while-extending-scala-class-with-Product-interface-to-overcome-limit-of-22-fields-in-spl-tp21787.html>
>>>>>> Sent from the Apache Spark User List mailing list archive
>>>>>> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Facing error while extending scala class with Product interface
to overcome limit of 22 fields in spark-shell
Posted by anamika gupta <an...@gmail.com>.
I am now getting the following error. I cross-checked my types and
corrected three of them i.e. r26-->String, r27-->Timestamp,
r28-->Timestamp. This error still persists.
scala> sc.textFile("/home/cdhuser/Desktop/Sdp_d.csv").map(_.split(",")).map
{ r =>
| val upto_time = sdf.parse(r(23).trim);
| calendar.setTime(upto_time);
| val r23 = new java.sql.Timestamp(upto_time.getTime)
| val insert_time = sdf.parse(r(27).trim)
| calendar.setTime(insert_time)
| val r27 = new java.sql.Timestamp(insert_time.getTime)
| val last_upd_time = sdf.parse(r(28).trim)
| calendar.setTime(last_upd_time)
| val r28 = new java.sql.Timestamp(last_upd_time.getTime)
| new sdp_d(r(0).trim.toInt, r(1).trim.toInt, r(2).trim,
r(3).trim.toInt, r(4).trim.toInt, r(5).trim, r(6).trim.toInt, r(7).trim,
r(8).trim.toDouble, r(9).trim.toDouble, r(10).trim, r(11).trim, r(12).trim,
r(13).trim, r(14).trim, r(15).trim, r(16).trim, r(17).trim, r(18).trim,
r(19).trim, r(20).trim, r(21).trim.toInt, r(22).trim, r23, r(24).trim,
r(25).trim, r(26).trim, r27, r28)
| }.registerAsTable("sdp_d")
<console>:26: error: type mismatch;
found : Int
required: Option[Int]
new sdp_d(r(0).trim.toInt, r(1).trim.toInt, r(2).trim,
r(3).trim.toInt, r(4).trim.toInt, r(5).trim, r(6).trim.toInt, r(7).trim,
r(8).trim.toDouble, r(9).trim.toDouble, r(10).trim, r(11).trim, r(12).trim,
r(13).trim, r(14).trim, r(15).trim, r(16).trim, r(17).trim, r(18).trim,
r(19).trim, r(20).trim, r(21).trim.toInt, r(22).trim, r23, r(24).trim,
r(25).trim, r(26).trim, r27, r28)
On Wed, Feb 25, 2015 at 2:32 PM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:
> It says sdp_d not found, since it is a class you need to instantiate it
> once. like:
>
> sc.textFile("derby.log").map(_.split(",")).map( r => {
> val upto_time = sdf.parse(r(23).trim);
> calendar.setTime(upto_time);
> val r23 = new java.sql.Timestamp(upto_time.getTime);
>
> val insert_time = sdf.parse(r(26).trim);
> calendar.setTime(insert_time);
> val r26 = new java.sql.Timestamp(insert_time.getTime);
>
> val last_upd_time = sdf.parse(r(27).trim);
> calendar.setTime(last_upd_time);
> val r27 = new java.sql.Timestamp(last_upd_time.getTime);
>
> *new* *sdp_d(r(0).trim.toInt, r(1).trim.toInt, r(2).trim,
> r(3).trim.toInt, r(4).trim.toInt, r(5).trim, r(6).trim.toInt, r(7).trim,
> r(8).trim.toDouble, r(9).trim.toDouble, r(10).trim, r(11).trim, r(12).trim,
> r(13).trim, r(14).trim, r(15).trim, r(16).trim, r(17).trim, r(18).trim,
> r(19).trim, r(20).trim, r(21).trim.toInt, r(22).trim, r23, r(24).trim,
> r(25).trim, r26, r27, r(28).trim)*
> }).registerAsTable("sdp")
>
> Thanks
> Best Regards
>
> On Wed, Feb 25, 2015 at 2:14 PM, anamika gupta <an...@gmail.com>
> wrote:
>
>> The link has proved helpful. I have been able to load data, register it
>> as a table and perform simple queries. Thanks Akhil !!
>>
>> Though, I still look forward to knowing where I was going wrong with my
>> previous technique of extending the Product Interface to overcome case
>> class's limit of 22 fields.
>>
>> On Wed, Feb 25, 2015 at 9:45 AM, anamika gupta <an...@gmail.com>
>> wrote:
>>
>>> Hi Akhil
>>>
>>> I guess it skipped my attention. I would definitely give it a try.
>>>
>>> While I would still like to know what is the issue with the way I have
>>> created schema?
>>>
>>> On Tue, Feb 24, 2015 at 4:35 PM, Akhil Das <ak...@sigmoidanalytics.com>
>>> wrote:
>>>
>>>> Did you happen to have a look at
>>>> https://spark.apache.org/docs/latest/sql-programming-guide.html#programmatically-specifying-the-schema
>>>>
>>>> Thanks
>>>> Best Regards
>>>>
>>>> On Tue, Feb 24, 2015 at 3:39 PM, anu <an...@gmail.com> wrote:
>>>>
>>>>> My issue is posted here on stack-overflow. What am I doing wrong here?
>>>>>
>>>>>
>>>>> http://stackoverflow.com/questions/28689186/facing-error-while-extending-scala-class-with-product-interface-to-overcome-limi
>>>>>
>>>>> ------------------------------
>>>>> View this message in context: Facing error while extending scala
>>>>> class with Product interface to overcome limit of 22 fields in spark-shell
>>>>> <http://apache-spark-user-list.1001560.n3.nabble.com/Facing-error-while-extending-scala-class-with-Product-interface-to-overcome-limit-of-22-fields-in-spl-tp21787.html>
>>>>> Sent from the Apache Spark User List mailing list archive
>>>>> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Facing error while extending scala class with Product interface
to overcome limit of 22 fields in spark-shell
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
It says sdp_d not found, since it is a class you need to instantiate it
once. like:
sc.textFile("derby.log").map(_.split(",")).map( r => {
val upto_time = sdf.parse(r(23).trim);
calendar.setTime(upto_time);
val r23 = new java.sql.Timestamp(upto_time.getTime);
val insert_time = sdf.parse(r(26).trim);
calendar.setTime(insert_time);
val r26 = new java.sql.Timestamp(insert_time.getTime);
val last_upd_time = sdf.parse(r(27).trim);
calendar.setTime(last_upd_time);
val r27 = new java.sql.Timestamp(last_upd_time.getTime);
*new* *sdp_d(r(0).trim.toInt, r(1).trim.toInt, r(2).trim,
r(3).trim.toInt, r(4).trim.toInt, r(5).trim, r(6).trim.toInt, r(7).trim,
r(8).trim.toDouble, r(9).trim.toDouble, r(10).trim, r(11).trim, r(12).trim,
r(13).trim, r(14).trim, r(15).trim, r(16).trim, r(17).trim, r(18).trim,
r(19).trim, r(20).trim, r(21).trim.toInt, r(22).trim, r23, r(24).trim,
r(25).trim, r26, r27, r(28).trim)*
}).registerAsTable("sdp")
Thanks
Best Regards
On Wed, Feb 25, 2015 at 2:14 PM, anamika gupta <an...@gmail.com>
wrote:
> The link has proved helpful. I have been able to load data, register it as
> a table and perform simple queries. Thanks Akhil !!
>
> Though, I still look forward to knowing where I was going wrong with my
> previous technique of extending the Product Interface to overcome case
> class's limit of 22 fields.
>
> On Wed, Feb 25, 2015 at 9:45 AM, anamika gupta <an...@gmail.com>
> wrote:
>
>> Hi Akhil
>>
>> I guess it skipped my attention. I would definitely give it a try.
>>
>> While I would still like to know what is the issue with the way I have
>> created schema?
>>
>> On Tue, Feb 24, 2015 at 4:35 PM, Akhil Das <ak...@sigmoidanalytics.com>
>> wrote:
>>
>>> Did you happen to have a look at
>>> https://spark.apache.org/docs/latest/sql-programming-guide.html#programmatically-specifying-the-schema
>>>
>>> Thanks
>>> Best Regards
>>>
>>> On Tue, Feb 24, 2015 at 3:39 PM, anu <an...@gmail.com> wrote:
>>>
>>>> My issue is posted here on stack-overflow. What am I doing wrong here?
>>>>
>>>>
>>>> http://stackoverflow.com/questions/28689186/facing-error-while-extending-scala-class-with-product-interface-to-overcome-limi
>>>>
>>>> ------------------------------
>>>> View this message in context: Facing error while extending scala class
>>>> with Product interface to overcome limit of 22 fields in spark-shell
>>>> <http://apache-spark-user-list.1001560.n3.nabble.com/Facing-error-while-extending-scala-class-with-Product-interface-to-overcome-limit-of-22-fields-in-spl-tp21787.html>
>>>> Sent from the Apache Spark User List mailing list archive
>>>> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>>>>
>>>
>>>
>>
>
Re: Facing error while extending scala class with Product interface
to overcome limit of 22 fields in spark-shell
Posted by anamika gupta <an...@gmail.com>.
The link has proved helpful. I have been able to load data, register it as
a table and perform simple queries. Thanks Akhil !!
Though, I still look forward to knowing where I was going wrong with my
previous technique of extending the Product Interface to overcome case
class's limit of 22 fields.
On Wed, Feb 25, 2015 at 9:45 AM, anamika gupta <an...@gmail.com>
wrote:
> Hi Akhil
>
> I guess it skipped my attention. I would definitely give it a try.
>
> While I would still like to know what is the issue with the way I have
> created schema?
>
> On Tue, Feb 24, 2015 at 4:35 PM, Akhil Das <ak...@sigmoidanalytics.com>
> wrote:
>
>> Did you happen to have a look at
>> https://spark.apache.org/docs/latest/sql-programming-guide.html#programmatically-specifying-the-schema
>>
>> Thanks
>> Best Regards
>>
>> On Tue, Feb 24, 2015 at 3:39 PM, anu <an...@gmail.com> wrote:
>>
>>> My issue is posted here on stack-overflow. What am I doing wrong here?
>>>
>>>
>>> http://stackoverflow.com/questions/28689186/facing-error-while-extending-scala-class-with-product-interface-to-overcome-limi
>>>
>>> ------------------------------
>>> View this message in context: Facing error while extending scala class
>>> with Product interface to overcome limit of 22 fields in spark-shell
>>> <http://apache-spark-user-list.1001560.n3.nabble.com/Facing-error-while-extending-scala-class-with-Product-interface-to-overcome-limit-of-22-fields-in-spl-tp21787.html>
>>> Sent from the Apache Spark User List mailing list archive
>>> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>>>
>>
>>
>
Re: Facing error while extending scala class with Product interface
to overcome limit of 22 fields in spark-shell
Posted by Petar Zecevic <pe...@gmail.com>.
I believe your class needs to be defined as a case class (as I answered
on SO)..
On 25.2.2015. 5:15, anamika gupta wrote:
> Hi Akhil
>
> I guess it skipped my attention. I would definitely give it a try.
>
> While I would still like to know what is the issue with the way I have
> created schema?
>
> On Tue, Feb 24, 2015 at 4:35 PM, Akhil Das <akhil@sigmoidanalytics.com
> <ma...@sigmoidanalytics.com>> wrote:
>
> Did you happen to have a look at
> https://spark.apache.org/docs/latest/sql-programming-guide.html#programmatically-specifying-the-schema
>
> Thanks
> Best Regards
>
> On Tue, Feb 24, 2015 at 3:39 PM, anu <anamika.guopta@gmail.com
> <ma...@gmail.com>> wrote:
>
> My issue is posted here on stack-overflow. What am I doing
> wrong here?
>
> http://stackoverflow.com/questions/28689186/facing-error-while-extending-scala-class-with-product-interface-to-overcome-limi
>
> ------------------------------------------------------------------------
> View this message in context: Facing error while extending
> scala class with Product interface to overcome limit of 22
> fields in spark-shell
> <http://apache-spark-user-list.1001560.n3.nabble.com/Facing-error-while-extending-scala-class-with-Product-interface-to-overcome-limit-of-22-fields-in-spl-tp21787.html>
> Sent from the Apache Spark User List mailing list archive
> <http://apache-spark-user-list.1001560.n3.nabble.com/> at
> Nabble.com.
>
>
>
Re: Facing error while extending scala class with Product interface
to overcome limit of 22 fields in spark-shell
Posted by anamika gupta <an...@gmail.com>.
Hi Akhil
I guess it skipped my attention. I would definitely give it a try.
While I would still like to know what is the issue with the way I have
created schema?
On Tue, Feb 24, 2015 at 4:35 PM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:
> Did you happen to have a look at
> https://spark.apache.org/docs/latest/sql-programming-guide.html#programmatically-specifying-the-schema
>
> Thanks
> Best Regards
>
> On Tue, Feb 24, 2015 at 3:39 PM, anu <an...@gmail.com> wrote:
>
>> My issue is posted here on stack-overflow. What am I doing wrong here?
>>
>>
>> http://stackoverflow.com/questions/28689186/facing-error-while-extending-scala-class-with-product-interface-to-overcome-limi
>>
>> ------------------------------
>> View this message in context: Facing error while extending scala class
>> with Product interface to overcome limit of 22 fields in spark-shell
>> <http://apache-spark-user-list.1001560.n3.nabble.com/Facing-error-while-extending-scala-class-with-Product-interface-to-overcome-limit-of-22-fields-in-spl-tp21787.html>
>> Sent from the Apache Spark User List mailing list archive
>> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>>
>
>
Re: Facing error while extending scala class with Product interface
to overcome limit of 22 fields in spark-shell
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Did you happen to have a look at
https://spark.apache.org/docs/latest/sql-programming-guide.html#programmatically-specifying-the-schema
Thanks
Best Regards
On Tue, Feb 24, 2015 at 3:39 PM, anu <an...@gmail.com> wrote:
> My issue is posted here on stack-overflow. What am I doing wrong here?
>
>
> http://stackoverflow.com/questions/28689186/facing-error-while-extending-scala-class-with-product-interface-to-overcome-limi
>
> ------------------------------
> View this message in context: Facing error while extending scala class
> with Product interface to overcome limit of 22 fields in spark-shell
> <http://apache-spark-user-list.1001560.n3.nabble.com/Facing-error-while-extending-scala-class-with-Product-interface-to-overcome-limit-of-22-fields-in-spl-tp21787.html>
> Sent from the Apache Spark User List mailing list archive
> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>