You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Maciej Szymkiewicz <ms...@gmail.com> on 2017/01/11 12:18:55 UTC
[PYSPARK] Python tests organization
Hi,
I can't help but wonder if there is any practical reason for keeping
monolithic test modules. These things are already pretty large (1500 -
2200 LOCs) and can only grow. Development aside, I assume that many
users use tests the same way as me, to check the intended behavior, and
largish loosely coupled modules make it harder than it should be.
If there's no rationale for that it could be a good time start thinking
about moving tests to packages and separating into modules reflecting
project structure.
--
Best,
Maciej
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [PYSPARK] Python tests organization
Posted by Reynold Xin <rx...@databricks.com>.
Yes absolutely.
On Wed, Jan 11, 2017 at 9:54 AM Saikat Kanjilal <sx...@hotmail.com> wrote:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Is it worth to come up with a proposal for this and float to dev?
>
>
>
>
>
>
>
>
>
>
> ------------------------------
>
>
> *From:* Reynold Xin <rx...@databricks.com>
>
>
> *Sent:* Wednesday, January 11, 2017 9:47 AM
>
>
> *To:* Maciej Szymkiewicz; Saikat Kanjilal; dev@spark.apache.org
>
>
> *Subject:* Re: [PYSPARK] Python tests organization
>
>
>
>
>
>
>
>
> It would be good to break them down a bit more, provided that we don't
> increase for example total runtime due to extra setup.
>
>
>
>
>
>
>
>
>
>
>
>
> On Wed, Jan 11, 2017 at 9:45 AM Saikat Kanjilal <sx...@hotmail.com>
> wrote:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Hello Maciej,
>
>
>
>
>
>
>
>
> If there's a jira available for this I'd like to help get this moving, let
> me know next steps.
>
>
>
>
>
>
>
>
> Thanks in advance.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> ------------------------------
>
>
>
>
>
>
>
>
> *From:* Maciej Szymkiewicz <ms...@gmail.com>
>
>
>
>
>
>
>
>
> *Sent:* Wednesday, January 11, 2017 4:18 AM
>
>
>
>
>
>
>
>
> *To:*
>
> dev@spark.apache.org
>
>
>
>
>
>
>
>
> *Subject:* [PYSPARK] Python tests organization
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Hi,
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> I can't help but wonder if there is any practical reason for keeping
>
>
>
>
>
>
>
>
> monolithic test modules. These things are already pretty large (1500 -
>
>
>
>
>
>
>
>
> 2200 LOCs) and can only grow. Development aside, I assume that many
>
>
>
>
>
>
>
>
> users use tests the same way as me, to check the intended behavior, and
>
>
>
>
>
>
>
>
> largish loosely coupled modules make it harder than it should be.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> If there's no rationale for that it could be a good time start thinking
>
>
>
>
>
>
>
>
> about moving tests to packages and separating into modules reflecting
>
>
>
>
>
>
>
>
> project structure.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
>
>
>
>
>
>
>
> Best,
>
>
>
>
>
>
>
>
> Maciej
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
>
>
>
>
>
>
>
>
> To unsubscribe e-mail:
>
> dev-unsubscribe@spark.apache.org
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
Re: [PYSPARK] Python tests organization
Posted by Saikat Kanjilal <sx...@hotmail.com>.
Is it worth to come up with a proposal for this and float to dev?
________________________________
From: Reynold Xin <rx...@databricks.com>
Sent: Wednesday, January 11, 2017 9:47 AM
To: Maciej Szymkiewicz; Saikat Kanjilal; dev@spark.apache.org
Subject: Re: [PYSPARK] Python tests organization
It would be good to break them down a bit more, provided that we don't increase for example total runtime due to extra setup.
On Wed, Jan 11, 2017 at 9:45 AM Saikat Kanjilal <sx...@hotmail.com>> wrote:
Hello Maciej,
If there's a jira available for this I'd like to help get this moving, let me know next steps.
Thanks in advance.
________________________________
From: Maciej Szymkiewicz <ms...@gmail.com>>
Sent: Wednesday, January 11, 2017 4:18 AM
To: dev@spark.apache.org<ma...@spark.apache.org>
Subject: [PYSPARK] Python tests organization
Hi,
I can't help but wonder if there is any practical reason for keeping
monolithic test modules. These things are already pretty large (1500 -
2200 LOCs) and can only grow. Development aside, I assume that many
users use tests the same way as me, to check the intended behavior, and
largish loosely coupled modules make it harder than it should be.
If there's no rationale for that it could be a good time start thinking
about moving tests to packages and separating into modules reflecting
project structure.
--
Best,
Maciej
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org<ma...@spark.apache.org>
Re: [PYSPARK] Python tests organization
Posted by Reynold Xin <rx...@databricks.com>.
It would be good to break them down a bit more, provided that we don't
increase for example total runtime due to extra setup.
On Wed, Jan 11, 2017 at 9:45 AM Saikat Kanjilal <sx...@hotmail.com> wrote:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Hello Maciej,
>
>
> If there's a jira available for this I'd like to help get this moving, let
> me know next steps.
>
>
> Thanks in advance.
>
>
>
>
>
>
>
>
>
>
>
>
> ------------------------------
>
>
> *From:* Maciej Szymkiewicz <ms...@gmail.com>
>
>
> *Sent:* Wednesday, January 11, 2017 4:18 AM
>
>
> *To:* dev@spark.apache.org
>
>
> *Subject:* [PYSPARK] Python tests organization
>
>
>
>
>
>
>
>
>
>
> Hi,
>
>
>
>
>
> I can't help but wonder if there is any practical reason for keeping
>
>
> monolithic test modules. These things are already pretty large (1500 -
>
>
> 2200 LOCs) and can only grow. Development aside, I assume that many
>
>
> users use tests the same way as me, to check the intended behavior, and
>
>
> largish loosely coupled modules make it harder than it should be.
>
>
>
>
>
> If there's no rationale for that it could be a good time start thinking
>
>
> about moving tests to packages and separating into modules reflecting
>
>
> project structure.
>
>
>
>
>
> --
>
>
> Best,
>
>
> Maciej
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
>
>
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
>
>
>
>
>
>
Re: [PYSPARK] Python tests organization
Posted by Saikat Kanjilal <sx...@hotmail.com>.
Maciej/Reynolds,
If its ok with you guys I can start working on a proposal and create a JIRA, let me know next steps.
Thanks in advance.
________________________________
From: Maciej Szymkiewicz <ms...@gmail.com>
Sent: Wednesday, January 11, 2017 10:14 AM
To: Saikat Kanjilal
Subject: Re: [PYSPARK] Python tests organization
Not yet, I want to see if there is any consensus about it. It is a lot of tedious work and I would be shame if someone started working on this just to get it dropped.
On 01/11/2017 06:44 PM, Saikat Kanjilal wrote:
Hello Maciej,
If there's a jira available for this I'd like to help get this moving, let me know next steps.
Thanks in advance.
________________________________
From: Maciej Szymkiewicz <ms...@gmail.com>
Sent: Wednesday, January 11, 2017 4:18 AM
To: dev@spark.apache.org<ma...@spark.apache.org>
Subject: [PYSPARK] Python tests organization
Hi,
I can't help but wonder if there is any practical reason for keeping
monolithic test modules. These things are already pretty large (1500 -
2200 LOCs) and can only grow. Development aside, I assume that many
users use tests the same way as me, to check the intended behavior, and
largish loosely coupled modules make it harder than it should be.
If there's no rationale for that it could be a good time start thinking
about moving tests to packages and separating into modules reflecting
project structure.
--
Best,
Maciej
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org<ma...@spark.apache.org>
--
Maciej Szymkiewicz
Re: [PYSPARK] Python tests organization
Posted by Maciej Szymkiewicz <ms...@gmail.com>.
Thanks Holden. If you have some spare time would you take a look at
https://github.com/apache/spark/pull/16534?
It is somewhat related to
https://issues.apache.org/jira/browse/SPARK-18777 (Return UDF objects
when registering from Python).
On 01/12/2017 07:34 PM, Holden Karau wrote:
> I'd be happy to help with reviewing Python test improvements. Maybe
> make an umbrella JIRA and do one sub components at a time?
>
> On Thu, Jan 12, 2017 at 12:20 PM Saikat Kanjilal <sxk1969@hotmail.com
> <ma...@hotmail.com>> wrote:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Following up, any thoughts on next steps for this?
>
>
>
>
>
>
>
>
>
>
>
> ------------------------------------------------------------------------
>
>
> *From:* Maciej Szymkiewicz <mszymkiewicz@gmail.com
> <ma...@gmail.com>>
>
>
>
> *Sent:* Wednesday, January 11, 2017 10:14 AM
>
>
> *To:* Saikat Kanjilal
>
>
> *Subject:* Re: [PYSPARK] Python tests organization
>
>
> Not yet, I want to see if there is any consensus about it. It is a
> lot of tedious work and I would be shame if someone started
> working on this just to get it dropped.
>
>
>
>
>
>
>
> On 01/11/2017 06:44 PM, Saikat Kanjilal wrote:
>
>
>
>
>>
>>
>>
>>
>> Hello Maciej,
>>
>>
>>
>> If there's a jira available for this I'd like to help get this
>> moving, let me know next steps.
>>
>>
>>
>> Thanks in advance.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>>
>> *From:* Maciej Szymkiewicz
>>
>> <ms...@gmail.com> <ma...@gmail.com>
>>
>>
>> *Sent:* Wednesday, January 11, 2017 4:18 AM
>>
>>
>> *To:*
>>
>> dev@spark.apache.org <ma...@spark.apache.org>
>>
>>
>> *Subject:* [PYSPARK] Python tests organization
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Hi,
>>
>>
>>
>>
>>
>> I can't help but wonder if there is any practical reason for keeping
>>
>>
>> monolithic test modules. These things are already pretty large
>> (1500 -
>>
>>
>> 2200 LOCs) and can only grow. Development aside, I assume that many
>>
>>
>> users use tests the same way as me, to check the intended
>> behavior, and
>>
>>
>> largish loosely coupled modules make it harder than it should be.
>>
>>
>>
>>
>>
>> If there's no rationale for that it could be a good time start
>> thinking
>>
>>
>> about moving tests to packages and separating into modules reflecting
>>
>>
>> project structure.
>>
>>
>>
>>
>>
>> --
>>
>>
>> Best,
>>
>>
>> Maciej
>>
>>
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>>
>>
>> To unsubscribe e-mail:
>>
>> dev-unsubscribe@spark.apache.org
>> <ma...@spark.apache.org>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
>
>
>
> --
> Maciej Szymkiewicz
>
>
>
--
Maciej Szymkiewicz
Re: [PYSPARK] Python tests organization
Posted by Saikat Kanjilal <sx...@hotmail.com>.
https://issues.apache.org/jira/browse/SPARK-19224
Maciej/Holden,
If its ok for I can come up with a proposal for reorganization and add the proposal to the JIRA as next steps before we break up the work?
Thanks
________________________________
From: Maciej Szymkiewicz <ms...@gmail.com>
Sent: Thursday, January 12, 2017 10:44 AM
To: dev@spark.apache.org
Subject: Re: [PYSPARK] Python tests organization
Sounds good, but it looks like JIRA is still down.
Personally I can look at sql.tests and see what can be done there. Depending on the resolution of https://issues.apache.org/jira/browse/SPARK-19160 I may have to adjust some tests anyway.
On 01/12/2017 07:36 PM, Saikat Kanjilal wrote:
Maciej? LGTM, what do you think? I can create a JIRA and drive this.
________________________________
From: Holden Karau <ho...@pigscanfly.ca>
Sent: Thursday, January 12, 2017 10:34 AM
To: Saikat Kanjilal; dev@spark.apache.org<ma...@spark.apache.org>
Subject: Re: [PYSPARK] Python tests organization
I'd be happy to help with reviewing Python test improvements. Maybe make an umbrella JIRA and do one sub components at a time?
On Thu, Jan 12, 2017 at 12:20 PM Saikat Kanjilal <sx...@hotmail.com>> wrote:
Following up, any thoughts on next steps for this?
________________________________
From: Maciej Szymkiewicz <ms...@gmail.com>>
Sent: Wednesday, January 11, 2017 10:14 AM
To: Saikat Kanjilal
Subject: Re: [PYSPARK] Python tests organization
Not yet, I want to see if there is any consensus about it. It is a lot of tedious work and I would be shame if someone started working on this just to get it dropped.
On 01/11/2017 06:44 PM, Saikat Kanjilal wrote:
Hello Maciej,
If there's a jira available for this I'd like to help get this moving, let me know next steps.
Thanks in advance.
________________________________
From: Maciej Szymkiewicz
<ms...@gmail.com>
Sent: Wednesday, January 11, 2017 4:18 AM
To:
dev@spark.apache.org<ma...@spark.apache.org>
Subject: [PYSPARK] Python tests organization
Hi,
I can't help but wonder if there is any practical reason for keeping
monolithic test modules. These things are already pretty large (1500 -
2200 LOCs) and can only grow. Development aside, I assume that many
users use tests the same way as me, to check the intended behavior, and
largish loosely coupled modules make it harder than it should be.
If there's no rationale for that it could be a good time start thinking
about moving tests to packages and separating into modules reflecting
project structure.
--
Best,
Maciej
---------------------------------------------------------------------
To unsubscribe e-mail:
dev-unsubscribe@spark.apache.org<ma...@spark.apache.org>
--
Maciej Szymkiewicz
--
Maciej Szymkiewicz
Re: [PYSPARK] Python tests organization
Posted by Maciej Szymkiewicz <ms...@gmail.com>.
Sounds good, but it looks like JIRA is still down.
Personally I can look at sql.tests and see what can be done there.
Depending on the resolution of
https://issues.apache.org/jira/browse/SPARK-19160 I may have to adjust
some tests anyway.
On 01/12/2017 07:36 PM, Saikat Kanjilal wrote:
>
> Maciej? LGTM, what do you think? I can create a JIRA and drive this.
>
>
>
> ------------------------------------------------------------------------
> *From:* Holden Karau <ho...@pigscanfly.ca>
> *Sent:* Thursday, January 12, 2017 10:34 AM
> *To:* Saikat Kanjilal; dev@spark.apache.org
> *Subject:* Re: [PYSPARK] Python tests organization
>
> I'd be happy to help with reviewing Python test improvements. Maybe
> make an umbrella JIRA and do one sub components at a time?
>
> On Thu, Jan 12, 2017 at 12:20 PM Saikat Kanjilal <sxk1969@hotmail.com
> <ma...@hotmail.com>> wrote:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Following up, any thoughts on next steps for this?
>
>
>
>
>
>
>
>
>
>
>
> ------------------------------------------------------------------------
>
>
> *From:* Maciej Szymkiewicz <mszymkiewicz@gmail.com
> <ma...@gmail.com>>
>
>
>
> *Sent:* Wednesday, January 11, 2017 10:14 AM
>
>
> *To:* Saikat Kanjilal
>
>
> *Subject:* Re: [PYSPARK] Python tests organization
>
>
> Not yet, I want to see if there is any consensus about it. It is a
> lot of tedious work and I would be shame if someone started
> working on this just to get it dropped.
>
>
>
>
>
>
>
> On 01/11/2017 06:44 PM, Saikat Kanjilal wrote:
>
>
>
>
>>
>>
>>
>>
>> Hello Maciej,
>>
>>
>>
>> If there's a jira available for this I'd like to help get this
>> moving, let me know next steps.
>>
>>
>>
>> Thanks in advance.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>>
>> *From:* Maciej Szymkiewicz
>>
>> <ms...@gmail.com> <ma...@gmail.com>
>>
>>
>> *Sent:* Wednesday, January 11, 2017 4:18 AM
>>
>>
>> *To:*
>>
>> dev@spark.apache.org <ma...@spark.apache.org>
>>
>>
>> *Subject:* [PYSPARK] Python tests organization
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Hi,
>>
>>
>>
>>
>>
>> I can't help but wonder if there is any practical reason for keeping
>>
>>
>> monolithic test modules. These things are already pretty large
>> (1500 -
>>
>>
>> 2200 LOCs) and can only grow. Development aside, I assume that many
>>
>>
>> users use tests the same way as me, to check the intended
>> behavior, and
>>
>>
>> largish loosely coupled modules make it harder than it should be.
>>
>>
>>
>>
>>
>> If there's no rationale for that it could be a good time start
>> thinking
>>
>>
>> about moving tests to packages and separating into modules reflecting
>>
>>
>> project structure.
>>
>>
>>
>>
>>
>> --
>>
>>
>> Best,
>>
>>
>> Maciej
>>
>>
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>>
>>
>> To unsubscribe e-mail:
>>
>> dev-unsubscribe@spark.apache.org
>> <ma...@spark.apache.org>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
>
>
>
> --
> Maciej Szymkiewicz
>
>
>
--
Maciej Szymkiewicz
Re: [PYSPARK] Python tests organization
Posted by Saikat Kanjilal <sx...@hotmail.com>.
Maciej? LGTM, what do you think? I can create a JIRA and drive this.
________________________________
From: Holden Karau <ho...@pigscanfly.ca>
Sent: Thursday, January 12, 2017 10:34 AM
To: Saikat Kanjilal; dev@spark.apache.org
Subject: Re: [PYSPARK] Python tests organization
I'd be happy to help with reviewing Python test improvements. Maybe make an umbrella JIRA and do one sub components at a time?
On Thu, Jan 12, 2017 at 12:20 PM Saikat Kanjilal <sx...@hotmail.com>> wrote:
Following up, any thoughts on next steps for this?
________________________________
From: Maciej Szymkiewicz <ms...@gmail.com>>
Sent: Wednesday, January 11, 2017 10:14 AM
To: Saikat Kanjilal
Subject: Re: [PYSPARK] Python tests organization
Not yet, I want to see if there is any consensus about it. It is a lot of tedious work and I would be shame if someone started working on this just to get it dropped.
On 01/11/2017 06:44 PM, Saikat Kanjilal wrote:
Hello Maciej,
If there's a jira available for this I'd like to help get this moving, let me know next steps.
Thanks in advance.
________________________________
From: Maciej Szymkiewicz
<ms...@gmail.com>
Sent: Wednesday, January 11, 2017 4:18 AM
To:
dev@spark.apache.org<ma...@spark.apache.org>
Subject: [PYSPARK] Python tests organization
Hi,
I can't help but wonder if there is any practical reason for keeping
monolithic test modules. These things are already pretty large (1500 -
2200 LOCs) and can only grow. Development aside, I assume that many
users use tests the same way as me, to check the intended behavior, and
largish loosely coupled modules make it harder than it should be.
If there's no rationale for that it could be a good time start thinking
about moving tests to packages and separating into modules reflecting
project structure.
--
Best,
Maciej
---------------------------------------------------------------------
To unsubscribe e-mail:
dev-unsubscribe@spark.apache.org<ma...@spark.apache.org>
--
Maciej Szymkiewicz
Re: [PYSPARK] Python tests organization
Posted by Holden Karau <ho...@pigscanfly.ca>.
I'd be happy to help with reviewing Python test improvements. Maybe make an
umbrella JIRA and do one sub components at a time?
On Thu, Jan 12, 2017 at 12:20 PM Saikat Kanjilal <sx...@hotmail.com>
wrote:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Following up, any thoughts on next steps for this?
>
>
>
>
>
>
>
>
>
>
> ------------------------------
>
>
> *From:* Maciej Szymkiewicz <ms...@gmail.com>
>
>
>
> *Sent:* Wednesday, January 11, 2017 10:14 AM
>
>
> *To:* Saikat Kanjilal
>
>
> *Subject:* Re: [PYSPARK] Python tests organization
>
>
> Not yet, I want to see if there is any consensus about it. It is a lot of
> tedious work and I would be shame if someone started working on this just
> to get it dropped.
>
>
>
>
>
>
>
> On 01/11/2017 06:44 PM, Saikat Kanjilal wrote:
>
>
>
>
>
>
>
>
> Hello Maciej,
>
>
> If there's a jira available for this I'd like to help get this moving, let
> me know next steps.
>
>
> Thanks in advance.
>
>
>
>
>
>
>
>
>
>
>
>
> ------------------------------
>
>
> *From:* Maciej Szymkiewicz
>
> <ms...@gmail.com> <ms...@gmail.com>
>
>
> *Sent:* Wednesday, January 11, 2017 4:18 AM
>
>
> *To:*
>
> dev@spark.apache.org
>
>
> *Subject:* [PYSPARK] Python tests organization
>
>
>
>
>
>
>
>
>
>
> Hi,
>
>
>
>
>
> I can't help but wonder if there is any practical reason for keeping
>
>
> monolithic test modules. These things are already pretty large (1500 -
>
>
> 2200 LOCs) and can only grow. Development aside, I assume that many
>
>
> users use tests the same way as me, to check the intended behavior, and
>
>
> largish loosely coupled modules make it harder than it should be.
>
>
>
>
>
> If there's no rationale for that it could be a good time start thinking
>
>
> about moving tests to packages and separating into modules reflecting
>
>
> project structure.
>
>
>
>
>
> --
>
>
> Best,
>
>
> Maciej
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
>
>
> To unsubscribe e-mail:
>
> dev-unsubscribe@spark.apache.org
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
> Maciej Szymkiewicz
>
>
>
>
Re: [PYSPARK] Python tests organization
Posted by Saikat Kanjilal <sx...@hotmail.com>.
Following up, any thoughts on next steps for this?
________________________________
From: Maciej Szymkiewicz <ms...@gmail.com>
Sent: Wednesday, January 11, 2017 10:14 AM
To: Saikat Kanjilal
Subject: Re: [PYSPARK] Python tests organization
Not yet, I want to see if there is any consensus about it. It is a lot of tedious work and I would be shame if someone started working on this just to get it dropped.
On 01/11/2017 06:44 PM, Saikat Kanjilal wrote:
Hello Maciej,
If there's a jira available for this I'd like to help get this moving, let me know next steps.
Thanks in advance.
________________________________
From: Maciej Szymkiewicz <ms...@gmail.com>
Sent: Wednesday, January 11, 2017 4:18 AM
To: dev@spark.apache.org<ma...@spark.apache.org>
Subject: [PYSPARK] Python tests organization
Hi,
I can't help but wonder if there is any practical reason for keeping
monolithic test modules. These things are already pretty large (1500 -
2200 LOCs) and can only grow. Development aside, I assume that many
users use tests the same way as me, to check the intended behavior, and
largish loosely coupled modules make it harder than it should be.
If there's no rationale for that it could be a good time start thinking
about moving tests to packages and separating into modules reflecting
project structure.
--
Best,
Maciej
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org<ma...@spark.apache.org>
--
Maciej Szymkiewicz
Re: [PYSPARK] Python tests organization
Posted by Saikat Kanjilal <sx...@hotmail.com>.
Hello Maciej,
If there's a jira available for this I'd like to help get this moving, let me know next steps.
Thanks in advance.
________________________________
From: Maciej Szymkiewicz <ms...@gmail.com>
Sent: Wednesday, January 11, 2017 4:18 AM
To: dev@spark.apache.org
Subject: [PYSPARK] Python tests organization
Hi,
I can't help but wonder if there is any practical reason for keeping
monolithic test modules. These things are already pretty large (1500 -
2200 LOCs) and can only grow. Development aside, I assume that many
users use tests the same way as me, to check the intended behavior, and
largish loosely coupled modules make it harder than it should be.
If there's no rationale for that it could be a good time start thinking
about moving tests to packages and separating into modules reflecting
project structure.
--
Best,
Maciej
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org