Coding in the Spark ml "ecosystem" why is everything private?!

[Thunder Stumpges <th...@gmail.com>]

Hi all,

I'm not sure if this belongs here in users or over in dev as I guess it's
somewhere in between. We have been starting to implement some machine
learning pipelines, and it seemed from the documentation that Spark had a
fairly well thought-out platform (see:
http://spark.apache.org/docs/1.6.1/ml-guide.html )

I liked the design of Transformers, Models, Estimators, Pipelines, etc.
However as soon as we began attempting to code our first ones, we began
running into one class or method after another that has been marked
private... Some examples are:

- SchemaUtils - (for validating schemas passed in and out, and adding
output columns to DataFrames)
- Loader / Saveable (traits / helpers for saving and loading models)
- Several classes under 'collection' namespace like OpenHashSet /
OpenHashMap
- All of the underlying linear algebra Breeze details
- Other classes specific to certain models. We are writing an alternative
LDA Optimizer / Trainer and everything under LDAUtils is private.

I'd like to ask what the expected approach is here. I see a few options,
none of which seem appropriate:

1. Implement everything in the org.apache.spark.* namespaces to match
package privates
    - will this even work in our own modules ?
    - we would be open to contributing some of our code back but not sure
the project wants it
2. Implement our own versions of all of these things.
   - lots of extra work for us, leads to unseen gotchas in implementations
and other unforseen issues
3. Copy classes into our namespace for use
   - duplicates code, leads to code diversion as the main code is kept up
to date.

Thanks in advance for any recommendations on this frustrating issue.
Thunder

Re: Coding in the Spark ml "ecosystem" why is everything private?!

[Thunder Stumpges <th...@gmail.com>]

I understand all that and can respect a team's desire not to have to
support many little internal details of a system; but at the same time, I
am talking about valuable aspects of a platform that make coding components
that work in an ecosystem viable. Let's put aside the OpenHashMap as I see
that supports your point and there are other libraries available.

However If I implement my own versions of Model save/load traits, How is
that even interoperable with the Pipeline class? And for schema validation
- the DataType class doesn't even expose the "equalsIgnoreNullability"...
so how would you check whether two types were the same?

There must be a balance between creating an environment that allows people
to relatively easily create components that work in said environment, and
having to "commit others' time to supporting it"...

It just feels like the balance is in general far to the "lock down" side of
the scale.
Thanks,
Thunder


On Mon, Aug 29, 2016 at 9:50 AM Sean Owen <so...@cloudera.com> wrote:

> If something isn't public, then it could change across even
> maintenance releases. Although you can indeed still access it in some
> cases by writing code in the same package, you're taking some risk
> that it will stop working across releases.
>
> If it's not public, the message is that you should build it yourself,
> yes. For example, OpenHashSet will never be meant to be reused. You
> can use your own from a library.
>
> If there's a clear opportunity to expose something cleanly you can
> bring it up for discussion. But it's never just a matter of making
> something public. Making it public means committing others' time to
> supporting it as-is for years. It would have to be worth it.
>
> On Mon, Aug 29, 2016 at 5:46 PM, Thunder Stumpges
> <th...@gmail.com> wrote:
> > Hi all,
> >
> > I'm not sure if this belongs here in users or over in dev as I guess it's
> > somewhere in between. We have been starting to implement some machine
> > learning pipelines, and it seemed from the documentation that Spark had a
> > fairly well thought-out platform (see:
> > http://spark.apache.org/docs/1.6.1/ml-guide.html )
> >
> > I liked the design of Transformers, Models, Estimators, Pipelines, etc.
> > However as soon as we began attempting to code our first ones, we began
> > running into one class or method after another that has been marked
> > private... Some examples are:
> >
> > - SchemaUtils - (for validating schemas passed in and out, and adding
> output
> > columns to DataFrames)
> > - Loader / Saveable (traits / helpers for saving and loading models)
> > - Several classes under 'collection' namespace like OpenHashSet /
> > OpenHashMap
> > - All of the underlying linear algebra Breeze details
> > - Other classes specific to certain models. We are writing an alternative
> > LDA Optimizer / Trainer and everything under LDAUtils is private.
> >
> > I'd like to ask what the expected approach is here. I see a few options,
> > none of which seem appropriate:
> >
> > 1. Implement everything in the org.apache.spark.* namespaces to match
> > package privates
> >     - will this even work in our own modules ?
> >     - we would be open to contributing some of our code back but not sure
> > the project wants it
> > 2. Implement our own versions of all of these things.
> >    - lots of extra work for us, leads to unseen gotchas in
> implementations
> > and other unforseen issues
> > 3. Copy classes into our namespace for use
> >    - duplicates code, leads to code diversion as the main code is kept
> up to
> > date.
> >
> > Thanks in advance for any recommendations on this frustrating issue.
> > Thunder
> >
>

Re: Coding in the Spark ml "ecosystem" why is everything private?!

[Sean Owen <so...@cloudera.com>]

If something isn't public, then it could change across even
maintenance releases. Although you can indeed still access it in some
cases by writing code in the same package, you're taking some risk
that it will stop working across releases.

If it's not public, the message is that you should build it yourself,
yes. For example, OpenHashSet will never be meant to be reused. You
can use your own from a library.

If there's a clear opportunity to expose something cleanly you can
bring it up for discussion. But it's never just a matter of making
something public. Making it public means committing others' time to
supporting it as-is for years. It would have to be worth it.

On Mon, Aug 29, 2016 at 5:46 PM, Thunder Stumpges
<th...@gmail.com> wrote:
> Hi all,
>
> I'm not sure if this belongs here in users or over in dev as I guess it's
> somewhere in between. We have been starting to implement some machine
> learning pipelines, and it seemed from the documentation that Spark had a
> fairly well thought-out platform (see:
> http://spark.apache.org/docs/1.6.1/ml-guide.html )
>
> I liked the design of Transformers, Models, Estimators, Pipelines, etc.
> However as soon as we began attempting to code our first ones, we began
> running into one class or method after another that has been marked
> private... Some examples are:
>
> - SchemaUtils - (for validating schemas passed in and out, and adding output
> columns to DataFrames)
> - Loader / Saveable (traits / helpers for saving and loading models)
> - Several classes under 'collection' namespace like OpenHashSet /
> OpenHashMap
> - All of the underlying linear algebra Breeze details
> - Other classes specific to certain models. We are writing an alternative
> LDA Optimizer / Trainer and everything under LDAUtils is private.
>
> I'd like to ask what the expected approach is here. I see a few options,
> none of which seem appropriate:
>
> 1. Implement everything in the org.apache.spark.* namespaces to match
> package privates
>     - will this even work in our own modules ?
>     - we would be open to contributing some of our code back but not sure
> the project wants it
> 2. Implement our own versions of all of these things.
>    - lots of extra work for us, leads to unseen gotchas in implementations
> and other unforseen issues
> 3. Copy classes into our namespace for use
>    - duplicates code, leads to code diversion as the main code is kept up to
> date.
>
> Thanks in advance for any recommendations on this frustrating issue.
> Thunder
>

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org