You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Maximilian Alber <al...@gmail.com> on 2014/10/07 15:13:57 UTC
Forced to use Solution Set in Step Function
Hi Flinksters!
I would like to use iterateDelta function. I don't need the solution set
inside the step function, because I generate a different values out of the
working set. Unfortunately the compiler of the development version doesn't
like that. Is there a workaround?
The code:
val residual_2a = residual_2 union
env.fromCollection(Seq(Vector.zeros(config.dimensions)))
val emptyDataSet = env.fromCollection[Vector](Seq())
val cumSum = emptyDataSet.iterateDelta(residual_2a, 1000000, Array("id")) {
(solutionset, workset) =>
val old_sum = workset filter {_.id == -1}
val current = workset filter (new RichFilterFunction[Vector]{
def filter(x: Vector) = x.id ==
(getIterationRuntimeContext.getSuperstepNumber)
})
val residual_2 = workset filter {_.id != -1}
val sum = VectorDataSet.add(old_sum, current)
(sum map (new RichMapFunction[Vector, Vector]{
def map(x: Vector) = new
Vector(getIterationRuntimeContext.getSuperstepNumber, x.values)
}),
residual_2 union sum)
}
The error:
org.apache.flink.compiler.CompilerException: Error: The step function does
not reference the solution set.
at
org.apache.flink.compiler.PactCompiler$GraphCreatingVisitor.postVisit(PactCompiler.java:868)
at
org.apache.flink.compiler.PactCompiler$GraphCreatingVisitor.postVisit(PactCompiler.java:622)
at
org.apache.flink.api.common.operators.DualInputOperator.accept(DualInputOperator.java:283)
at
org.apache.flink.api.common.operators.SingleInputOperator.accept(SingleInputOperator.java:202)
at
org.apache.flink.api.common.operators.GenericDataSinkBase.accept(GenericDataSinkBase.java:286)
at org.apache.flink.api.common.Plan.accept(Plan.java:281)
at org.apache.flink.compiler.PactCompiler.compile(PactCompiler.java:517)
at org.apache.flink.compiler.PactCompiler.compile(PactCompiler.java:466)
at org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:196)
at org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:209)
at org.apache.flink.client.program.Client.run(Client.java:285)
at org.apache.flink.client.program.Client.run(Client.java:230)
at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:347)
at org.apache.flink.client.CliFrontend.run(CliFrontend.java:334)
at
org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1001)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1025)
Thanks!
Cheers,
Max
Re: Forced to use Solution Set in Step Function
Posted by Maximilian Alber <al...@gmail.com>.
It should be no problem to make it part of the loop or getting residuals_2
into the workset, it is just a bit inefficient.
No, that would not make sense. old_sum and current are two datasets with
just one vector.
I think putting residuals_2 into the workset and then filter for current
and old_sum is the best workaround.
On Tue, Oct 14, 2014 at 5:15 PM, Stephan Ewen <se...@apache.org> wrote:
> That is an interesting case. Everything that is loop invariant is computed
> once outside the loop. You are looking for a way to make this part of the
> loop.
>
> Can you try making the filter part of the "VectorDataSet.add(old_sum,
> current)" operation?
>
> On Tue, Oct 14, 2014 at 5:05 PM, Maximilian Alber <
> alber.maximilian@gmail.com> wrote:
>
>> The deltaiteration calculates the cumulative sum (prefix sum) of
>> residual_2.
>>
>> I'm not sure if got you idea. I could add residual_2 to the workset. But
>> in that case I need to separate the "old_sum"(the sum up to now) and
>> residual_2 each iteration. Actually I had that before, then I tried the
>> Scala closure to clean up the code.
>>
>> On Tue, Oct 14, 2014 at 4:57 PM, Fabian Hueske <fh...@apache.org>
>> wrote:
>>
>>> Jep, I see you point.
>>> Conceptually, all data that changes and affects the result of an
>>> iteration should be part of the workset.
>>> Hence, the model kind of assumes that the datum "superStepNumber" should
>>> be part of the workset.
>>>
>>> I am not familiar with your application, but would it make sense to add
>>> the number as an additional attribute to the workset data set and increase
>>> it manually?
>>>
>>> 2014-10-14 16:45 GMT+02:00 Maximilian Alber <al...@gmail.com>
>>> :
>>>
>>>> Ok, sounds true, but somehow I would like to execute it inside of it.
>>>> So I probably need to do some nonsense work to make it part of it?
>>>>
>>>> On Tue, Oct 14, 2014 at 4:36 PM, Aljoscha Krettek <al...@apache.org>
>>>> wrote:
>>>>
>>>>> Dammit you beat me to it. But yes, this is exactly what I was just
>>>>> writing.
>>>>>
>>>>> On Tue, Oct 14, 2014 at 4:35 PM, Fabian Hueske <fh...@apache.org>
>>>>> wrote:
>>>>> > Hi,
>>>>> >
>>>>> > I'm not super familiar with the iterations, but my guess would be
>>>>> that the
>>>>> > filter is not evaluated as part of the iteration.
>>>>> > Since it is not connect to the workset, the filter is not part of
>>>>> the loop
>>>>> > and evaluated once outside where no superset number is available.
>>>>> > I guess, moving the filter outside of the loop gives the same error.
>>>>> >
>>>>> > Cheers, Fabian
>>>>> >
>>>>> >
>>>>> >
>>>>> > 2014-10-14 16:18 GMT+02:00 Maximilian Alber <
>>>>> alber.maximilian@gmail.com>:
>>>>> >>
>>>>> >> Hmm or maybe not. With this code I get some strange error:
>>>>> >>
>>>>> >> def createPlan_find_center(env: ExecutionEnvironment) = {
>>>>> >> val X = env readTextFile config.xFile map
>>>>> >> {Vector.parseFromString(config.dimensions, _)};
>>>>> >> val residual = env readTextFile config.yFile map
>>>>> >> {Vector.parseFromString(_)};
>>>>> >> val randoms = env readTextFile config.randomFile map
>>>>> >> {Vector.parseFromString(_)}
>>>>> >>
>>>>> >> val residual_2 = residual * residual
>>>>> >> val ys = (residual_2 sumV) * (randoms filter {_.id == 0})
>>>>> >>
>>>>> >> val emptyDataSet = env.fromCollection[Vector](Seq())
>>>>> >> val sumVector =
>>>>> env.fromCollection(Seq(Vector.zeros(config.dimensions)))
>>>>> >> val cumSum = emptyDataSet.iterateDelta(sumVector, config.N,
>>>>> Array("id")) {
>>>>> >> (solutionset, old_sum) =>
>>>>> >> val current = residual_2 filter (new RichFilterFunction[Vector]{
>>>>> >> def filter(x: Vector) = x.id ==
>>>>> >> (getIterationRuntimeContext.getSuperstepNumber)
>>>>> >> })
>>>>> >> val sum = VectorDataSet.add(old_sum, current)
>>>>> >>
>>>>> >> (sum map (new RichMapFunction[Vector, Vector]{
>>>>> >> def map(x: Vector) = new
>>>>> >> Vector(getIterationRuntimeContext.getSuperstepNumber, x.values)
>>>>> >> }),
>>>>> >> sum)
>>>>> >> }
>>>>> >>
>>>>> >> Error:
>>>>> >> 10/14/2014 15:57:35: Job execution switched to status RUNNING
>>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to
>>>>> SCHEDULED
>>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to
>>>>> DEPLOYING
>>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to SCHEDULED
>>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to DEPLOYING
>>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>>>>> (/tmp/tmpBhOsLd) -
>>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>>>> switched to
>>>>> >> SCHEDULED
>>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>>>>> (/tmp/tmpBhOsLd) -
>>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>>>> switched to
>>>>> >> DEPLOYING
>>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to RUNNING
>>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
>>>>> >> Iteration)) (1/1) switched to SCHEDULED
>>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
>>>>> >> Iteration)) (1/1) switched to DEPLOYING
>>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>>>>> (/tmp/tmpBhOsLd) -
>>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>>>> switched to
>>>>> >> RUNNING
>>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to RUNNING
>>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
>>>>> >> Iteration)) (1/1) switched to RUNNING
>>>>> >> 10/14/2014 15:57:35: CHAIN
>>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>>> Map
>>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to SCHEDULED
>>>>> >> 10/14/2014 15:57:35: CHAIN
>>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>>> Map
>>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to DEPLOYING
>>>>> >> 10/14/2014 15:57:36: CHAIN
>>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>>> Map
>>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>>>>> SCHEDULED
>>>>> >> 10/14/2014 15:57:36: CHAIN
>>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>>> Map
>>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>>>>> DEPLOYING
>>>>> >> 10/14/2014 15:57:36: CHAIN
>>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>>> Map
>>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to RUNNING
>>>>> >> 10/14/2014 15:57:36: CHAIN
>>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>>> Map
>>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>>>>> RUNNING
>>>>> >> 10/14/2014 15:57:36: CHAIN
>>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>>> Map
>>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to FAILED
>>>>> >> java.lang.IllegalStateException: This stub is not part of an
>>>>> iteration
>>>>> >> step function.
>>>>> >> at
>>>>> >>
>>>>> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
>>>>> >> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
>>>>> >> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
>>>>> >> at
>>>>> >>
>>>>> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
>>>>> >> at
>>>>> >>
>>>>> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
>>>>> >> at
>>>>> >>
>>>>> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
>>>>> >> at
>>>>> >>
>>>>> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
>>>>> >> at
>>>>> >>
>>>>> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
>>>>> >> at
>>>>> >>
>>>>> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
>>>>> >> at
>>>>> >>
>>>>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
>>>>> >> at
>>>>> >>
>>>>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
>>>>> >> at
>>>>> >>
>>>>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
>>>>> >> at java.lang.Thread.run(Thread.java:745)
>>>>> >>
>>>>> >> 10/14/2014 15:57:36: Job execution switched to status FAILING
>>>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to CANCELING
>>>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
>>>>> (/tmp/tmpBhOsLd) -
>>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>>>> switched to
>>>>> >> CANCELING
>>>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to
>>>>> CANCELING
>>>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed Delta
>>>>> >> Iteration)) (1/1) switched to CANCELING
>>>>> >> 10/14/2014 15:57:36: Map
>>>>> (org.apache.flink.api.scala.DataSet$$anon$1)
>>>>> >> (1/1) switched to CANCELED
>>>>> >> 10/14/2014 15:57:36: DataSink(TextOutputFormat (/tmp/tmplSYJ7S) -
>>>>> UTF-8)
>>>>> >> (1/1) switched to CANCELED
>>>>> >> 10/14/2014 15:57:36: Sync (WorksetIteration (Unnamed Delta
>>>>> Iteration))
>>>>> >> (1/1) switched to CANCELED
>>>>> >> 10/14/2014 15:57:36: CHAIN
>>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>>> Map
>>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>>>>> CANCELING
>>>>> >> 10/14/2014 15:57:36: Map (bumpboost.BumpBoost$$anonfun$8$$anon$2)
>>>>> (1/1)
>>>>> >> switched to CANCELED
>>>>> >> 10/14/2014 15:57:36: SolutionSet Delta (1/1) switched to CANCELED
>>>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to CANCELED
>>>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
>>>>> (/tmp/tmpBhOsLd) -
>>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>>>> switched to
>>>>> >> CANCELED
>>>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to
>>>>> CANCELED
>>>>> >> 10/14/2014 15:57:36: CHAIN
>>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>>> Map
>>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>>>>> CANCELED
>>>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed Delta
>>>>> >> Iteration)) (1/1) switched to CANCELED
>>>>> >> 10/14/2014 15:57:36: Job execution switched to status FAILED
>>>>> >> Error: The program execution failed:
>>>>> java.lang.IllegalStateException: This
>>>>> >> stub is not part of an iteration step function.
>>>>> >> at
>>>>> >>
>>>>> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
>>>>> >> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
>>>>> >> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
>>>>> >> at
>>>>> >>
>>>>> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
>>>>> >> at
>>>>> >>
>>>>> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
>>>>> >> at
>>>>> >>
>>>>> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
>>>>> >> at
>>>>> >>
>>>>> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
>>>>> >> at
>>>>> >>
>>>>> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
>>>>> >> at
>>>>> >>
>>>>> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
>>>>> >> at
>>>>> >>
>>>>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
>>>>> >> at
>>>>> >>
>>>>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
>>>>> >> at
>>>>> >>
>>>>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
>>>>> >> at java.lang.Thread.run(Thread.java:745)
>>>>> >>
>>>>> >> On Tue, Oct 14, 2014 at 3:32 PM, Maximilian Alber
>>>>> >> <al...@gmail.com> wrote:
>>>>> >>>
>>>>> >>> Should work now.
>>>>> >>> Cheers
>>>>> >>>
>>>>> >>> On Fri, Oct 10, 2014 at 3:38 PM, Maximilian Alber
>>>>> >>> <al...@gmail.com> wrote:
>>>>> >>>>
>>>>> >>>> Ok, thanks.
>>>>> >>>> Please let me know when it is fixed.
>>>>> >>>>
>>>>> >>>> Cheers
>>>>> >>>> Max
>>>>> >>>>
>>>>> >>>> On Fri, Oct 10, 2014 at 1:34 PM, Stephan Ewen <se...@apache.org>
>>>>> wrote:
>>>>> >>>>>
>>>>> >>>>> Thank you, I will look into that...
>>>>> >>>>
>>>>> >>>>
>>>>> >>>
>>>>> >>
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Forced to use Solution Set in Step Function
Posted by Stephan Ewen <se...@apache.org>.
Good to heat that it worked. No need to try out the other version, the same
fix is in there...
On Tue, Nov 11, 2014 at 2:07 PM, Maximilian Alber <
alber.maximilian@gmail.com> wrote:
> Hi Stephan!
>
> It worked with the development snapshot. I never tried it with the new
> release. If you want me to try it out, just say so.
>
> Cheers,
> Max
>
> On Mon, Nov 10, 2014 at 5:13 PM, Stephan Ewen <se...@apache.org> wrote:
>
>> Hi!
>>
>> The 0.7.0 release should have fixed that problem. Have you had a chance
>> to try that out?
>>
>> Greetings,
>> Stephan
>>
>>
>> On Wed, Oct 15, 2014 at 3:43 PM, Maximilian Alber <
>> alber.maximilian@gmail.com> wrote:
>>
>>> @Aljoscha Sorry, I just tried my workaround. There are some minor
>>> conceptual bugs (caused by the id's...). I attached the new version.
>>> Unfortunately there the compiler breaks. An issue is already open.
>>>
>>> On Wed, Oct 15, 2014 at 11:53 AM, Maximilian Alber <
>>> alber.maximilian@gmail.com> wrote:
>>>
>>>> Ok.
>>>>
>>>> Here is a input variant:
>>>> flink run -v bump_boost-0.1.jar -c bumpboost.Job x_file=X y_file=Y
>>>> out_file=/tmp/tmpnWYamw random_file=random_file dimensions=1 N=100
>>>> width_candidates_file=/tmp/tmpTJ4LDh iterations=30 multi_bump_boost=0
>>>> gradient_descent_iterations=30 cache=False min_width=-4 max_width=6
>>>> min_width_update=1e-08 max_width_update=10
>>>>
>>>> width_candidates_file is not needed by now. X, Y and random_file are
>>>> attached.
>>>>
>>>> If you have problems, running it, let me know!
>>>> Thanks!
>>>>
>>>> On Tue, Oct 14, 2014 at 10:40 PM, Aljoscha Krettek <aljoscha@apache.org
>>>> > wrote:
>>>>
>>>>> Transferring to variables: Unfortunately not possible right now but we
>>>>> are working on it.
>>>>>
>>>>> On Tue, Oct 14, 2014 at 8:53 PM, Maximilian Alber
>>>>> <al...@gmail.com> wrote:
>>>>> > @ Stephan: Thanks! So I gonna switch!
>>>>> >
>>>>> > Sorry, my bad. I will provide you some sample by tomorrow morning.
>>>>> >
>>>>> > Yes. Workaround, because I cannot transfer them into variables, can
>>>>> I by now
>>>>> > (or will I ever)?
>>>>> >
>>>>> > Maybe some explanation to my solution:
>>>>> > - X is for my a matrix of shape (N, d). Modeled in Flink as dataset
>>>>> of
>>>>> > vectors. Each Vector has an ID which is the row number and an array
>>>>> with
>>>>> > numbers, the actual row.
>>>>> > - Y is for my a matrix of shape (N, 1) thus actually a column-vector.
>>>>> > - old_sum is either a scalar if d == 1 or a row-vector aka matrix of
>>>>> shape
>>>>> > (1, N) or a Dataset with one Vector. (By now I have the convention
>>>>> to give
>>>>> > id -1 to them, comes from a former workaround...)
>>>>> >
>>>>> > The whole ID story comes from the fact that I need to know which
>>>>> stuff
>>>>> > belongs together in mathematical operations (see my zip function).
>>>>> You can
>>>>> > look that up in util.scala, that's kind of my math library. I don't
>>>>> want to
>>>>> > imagine the mess in Java :)
>>>>> >
>>>>> > Cheers
>>>>> > Max
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Tue, Oct 14, 2014 at 6:28 PM, Aljoscha Krettek <
>>>>> aljoscha@apache.org>
>>>>> > wrote:
>>>>> >>
>>>>> >> Could you maybe also give some examples for the input expected by
>>>>> your
>>>>> >> program?
>>>>> >>
>>>>> >> Also, the residual DataSet contains several Vectors while the sum
>>>>> (or
>>>>> >> old_sum) DataSet is always only contains 1 Vector. Correct?
>>>>> >>
>>>>> >> On Tue, Oct 14, 2014 at 6:04 PM, Stephan Ewen <se...@apache.org>
>>>>> wrote:
>>>>> >> > BTW: The current master allows you to not join with the solution
>>>>> set,
>>>>> >> > and
>>>>> >> > only use it to accumulate data.
>>>>> >> >
>>>>> >> > On Tue, Oct 14, 2014 at 5:29 PM, Maximilian Alber
>>>>> >> > <al...@gmail.com> wrote:
>>>>> >> >>
>>>>> >> >> Ok, that's possible too.
>>>>> >> >>
>>>>> >> >> VectorDataSet is just scala magic to ease my life (See below).
>>>>> If you
>>>>> >> >> want
>>>>> >> >> to take a look, I appended the package. The main code is in
>>>>> >> >> BumpBoost.scala.
>>>>> >> >> In util.scala is the vector stuff.
>>>>> >> >> Thanks!
>>>>> >> >>
>>>>> >> >> class VectorDataSet(X: DataSet[Vector]){
>>>>> >> >> def + (Y: DataSet[Vector]) = VectorDataSet.add(X, Y)
>>>>> >> >> def - (Y: DataSet[Vector]) = VectorDataSet.subt(X, Y)
>>>>> >> >> def * (Y: DataSet[Vector]) = VectorDataSet.mult(X, Y)
>>>>> >> >> def / (Y: DataSet[Vector]) = VectorDataSet.div(X, Y)
>>>>> >> >>
>>>>> >> >> def zip(Y: DataSet[Vector]) = VectorDataSet.zip(X, Y)
>>>>> >> >> def dot(Y: DataSet[Vector]) = VectorDataSet.dot(X, Y)
>>>>> >> >> def sumV() = VectorDataSet.sumV(X)
>>>>> >> >> }
>>>>> >> >> object VectorDataSet {
>>>>> >> >> def add(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2
>>>>> map {x =>
>>>>> >> >> x._1 + x._2}
>>>>> >> >> def subt(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2
>>>>> map {x
>>>>> >> >> =>
>>>>> >> >> x._1 - x._2}
>>>>> >> >> def mult(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2
>>>>> map {x
>>>>> >> >> =>
>>>>> >> >> x._1 * x._2}
>>>>> >> >> def div(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2
>>>>> map {x =>
>>>>> >> >> x._1 / x._2}
>>>>> >> >>
>>>>> >> >> def zip(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 join X2
>>>>> where
>>>>> >> >> "id"
>>>>> >> >> equalTo "id"
>>>>> >> >> def dot(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 * X2 sumV
>>>>> >> >> def sumV(X1: DataSet[Vector]) = X1 reduce {_ + _}
>>>>> >> >>
>>>>> >> >> implicit def vectorDataSetWrapper(ds: DataSet[Vector]) = new
>>>>> >> >> VectorDataSet(ds)
>>>>> >> >> }
>>>>> >> >>
>>>>> >> >>
>>>>> >> >> On Tue, Oct 14, 2014 at 5:21 PM, Aljoscha Krettek <
>>>>> aljoscha@apache.org>
>>>>> >> >> wrote:
>>>>> >> >>>
>>>>> >> >>> Maybe you could use the residual_2 data set as a broadcast
>>>>> dataset.
>>>>> >> >>> i.e. make in available in the operation that adds the residual
>>>>> for the
>>>>> >> >>> current iteration number to the old_sum. (I'm not sure what the
>>>>> >> >>> VectorDataSet.add() method does here). If you gave me the
>>>>> complete
>>>>> >> >>> code I could try finding an elegant solution to that problem.
>>>>> >> >>>
>>>>> >> >>> On Tue, Oct 14, 2014 at 5:15 PM, Stephan Ewen <sewen@apache.org
>>>>> >
>>>>> >> >>> wrote:
>>>>> >> >>> > That is an interesting case. Everything that is loop
>>>>> invariant is
>>>>> >> >>> > computed
>>>>> >> >>> > once outside the loop. You are looking for a way to make this
>>>>> part
>>>>> >> >>> > of
>>>>> >> >>> > the
>>>>> >> >>> > loop.
>>>>> >> >>> >
>>>>> >> >>> > Can you try making the filter part of the
>>>>> >> >>> > "VectorDataSet.add(old_sum,
>>>>> >> >>> > current)" operation?
>>>>> >> >>> >
>>>>> >> >>> > On Tue, Oct 14, 2014 at 5:05 PM, Maximilian Alber
>>>>> >> >>> > <al...@gmail.com> wrote:
>>>>> >> >>> >>
>>>>> >> >>> >> The deltaiteration calculates the cumulative sum (prefix
>>>>> sum) of
>>>>> >> >>> >> residual_2.
>>>>> >> >>> >>
>>>>> >> >>> >> I'm not sure if got you idea. I could add residual_2 to the
>>>>> >> >>> >> workset.
>>>>> >> >>> >> But
>>>>> >> >>> >> in that case I need to separate the "old_sum"(the sum up to
>>>>> now)
>>>>> >> >>> >> and
>>>>> >> >>> >> residual_2 each iteration. Actually I had that before, then
>>>>> I tried
>>>>> >> >>> >> the
>>>>> >> >>> >> Scala closure to clean up the code.
>>>>> >> >>> >>
>>>>> >> >>> >> On Tue, Oct 14, 2014 at 4:57 PM, Fabian Hueske <
>>>>> fhueske@apache.org>
>>>>> >> >>> >> wrote:
>>>>> >> >>> >>>
>>>>> >> >>> >>> Jep, I see you point.
>>>>> >> >>> >>> Conceptually, all data that changes and affects the result
>>>>> of an
>>>>> >> >>> >>> iteration should be part of the workset.
>>>>> >> >>> >>> Hence, the model kind of assumes that the datum
>>>>> "superStepNumber"
>>>>> >> >>> >>> should
>>>>> >> >>> >>> be part of the workset.
>>>>> >> >>> >>>
>>>>> >> >>> >>> I am not familiar with your application, but would it make
>>>>> sense
>>>>> >> >>> >>> to
>>>>> >> >>> >>> add
>>>>> >> >>> >>> the number as an additional attribute to the workset data
>>>>> set and
>>>>> >> >>> >>> increase
>>>>> >> >>> >>> it manually?
>>>>> >> >>> >>>
>>>>> >> >>> >>> 2014-10-14 16:45 GMT+02:00 Maximilian Alber
>>>>> >> >>> >>> <al...@gmail.com>:
>>>>> >> >>> >>>>
>>>>> >> >>> >>>> Ok, sounds true, but somehow I would like to execute it
>>>>> inside of
>>>>> >> >>> >>>> it. So
>>>>> >> >>> >>>> I probably need to do some nonsense work to make it part
>>>>> of it?
>>>>> >> >>> >>>>
>>>>> >> >>> >>>> On Tue, Oct 14, 2014 at 4:36 PM, Aljoscha Krettek
>>>>> >> >>> >>>> <al...@apache.org>
>>>>> >> >>> >>>> wrote:
>>>>> >> >>> >>>>>
>>>>> >> >>> >>>>> Dammit you beat me to it. But yes, this is exactly what I
>>>>> was
>>>>> >> >>> >>>>> just
>>>>> >> >>> >>>>> writing.
>>>>> >> >>> >>>>>
>>>>> >> >>> >>>>> On Tue, Oct 14, 2014 at 4:35 PM, Fabian Hueske
>>>>> >> >>> >>>>> <fh...@apache.org>
>>>>> >> >>> >>>>> wrote:
>>>>> >> >>> >>>>> > Hi,
>>>>> >> >>> >>>>> >
>>>>> >> >>> >>>>> > I'm not super familiar with the iterations, but my
>>>>> guess would
>>>>> >> >>> >>>>> > be
>>>>> >> >>> >>>>> > that the
>>>>> >> >>> >>>>> > filter is not evaluated as part of the iteration.
>>>>> >> >>> >>>>> > Since it is not connect to the workset, the filter is
>>>>> not part
>>>>> >> >>> >>>>> > of
>>>>> >> >>> >>>>> > the
>>>>> >> >>> >>>>> > loop
>>>>> >> >>> >>>>> > and evaluated once outside where no superset number is
>>>>> >> >>> >>>>> > available.
>>>>> >> >>> >>>>> > I guess, moving the filter outside of the loop gives
>>>>> the same
>>>>> >> >>> >>>>> > error.
>>>>> >> >>> >>>>> >
>>>>> >> >>> >>>>> > Cheers, Fabian
>>>>> >> >>> >>>>> >
>>>>> >> >>> >>>>> >
>>>>> >> >>> >>>>> >
>>>>> >> >>> >>>>> > 2014-10-14 16:18 GMT+02:00 Maximilian Alber
>>>>> >> >>> >>>>> > <al...@gmail.com>:
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >> Hmm or maybe not. With this code I get some strange
>>>>> error:
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >> def createPlan_find_center(env: ExecutionEnvironment)
>>>>> = {
>>>>> >> >>> >>>>> >> val X = env readTextFile config.xFile map
>>>>> >> >>> >>>>> >> {Vector.parseFromString(config.dimensions, _)};
>>>>> >> >>> >>>>> >> val residual = env readTextFile config.yFile map
>>>>> >> >>> >>>>> >> {Vector.parseFromString(_)};
>>>>> >> >>> >>>>> >> val randoms = env readTextFile config.randomFile map
>>>>> >> >>> >>>>> >> {Vector.parseFromString(_)}
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >> val residual_2 = residual * residual
>>>>> >> >>> >>>>> >> val ys = (residual_2 sumV) * (randoms filter {_.id ==
>>>>> 0})
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >> val emptyDataSet = env.fromCollection[Vector](Seq())
>>>>> >> >>> >>>>> >> val sumVector =
>>>>> >> >>> >>>>> >>
>>>>> env.fromCollection(Seq(Vector.zeros(config.dimensions)))
>>>>> >> >>> >>>>> >> val cumSum = emptyDataSet.iterateDelta(sumVector,
>>>>> config.N,
>>>>> >> >>> >>>>> >> Array("id")) {
>>>>> >> >>> >>>>> >> (solutionset, old_sum) =>
>>>>> >> >>> >>>>> >> val current = residual_2 filter (new
>>>>> >> >>> >>>>> >> RichFilterFunction[Vector]{
>>>>> >> >>> >>>>> >> def filter(x: Vector) = x.id ==
>>>>> >> >>> >>>>> >> (getIterationRuntimeContext.getSuperstepNumber)
>>>>> >> >>> >>>>> >> })
>>>>> >> >>> >>>>> >> val sum = VectorDataSet.add(old_sum, current)
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >> (sum map (new RichMapFunction[Vector, Vector]{
>>>>> >> >>> >>>>> >> def map(x: Vector) = new
>>>>> >> >>> >>>>> >> Vector(getIterationRuntimeContext.getSuperstepNumber,
>>>>> >> >>> >>>>> >> x.values)
>>>>> >> >>> >>>>> >> }),
>>>>> >> >>> >>>>> >> sum)
>>>>> >> >>> >>>>> >> }
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >> Error:
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: Job execution switched to status
>>>>> RUNNING
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1)
>>>>> switched to
>>>>> >> >>> >>>>> >> SCHEDULED
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1)
>>>>> switched to
>>>>> >> >>> >>>>> >> DEPLOYING
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to
>>>>> >> >>> >>>>> >> SCHEDULED
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to
>>>>> >> >>> >>>>> >> DEPLOYING
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>>>>> >> >>> >>>>> >> (/tmp/tmpBhOsLd) -
>>>>> >> >>> >>>>> >> UTF-8) -> Map
>>>>> (org.apache.flink.api.scala.DataSet$$anon$1)
>>>>> >> >>> >>>>> >> (1/1)
>>>>> >> >>> >>>>> >> switched to
>>>>> >> >>> >>>>> >> SCHEDULED
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>>>>> >> >>> >>>>> >> (/tmp/tmpBhOsLd) -
>>>>> >> >>> >>>>> >> UTF-8) -> Map
>>>>> (org.apache.flink.api.scala.DataSet$$anon$1)
>>>>> >> >>> >>>>> >> (1/1)
>>>>> >> >>> >>>>> >> switched to
>>>>> >> >>> >>>>> >> DEPLOYING
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1)
>>>>> switched to
>>>>> >> >>> >>>>> >> RUNNING
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration
>>>>> (Unnamed
>>>>> >> >>> >>>>> >> Delta
>>>>> >> >>> >>>>> >> Iteration)) (1/1) switched to SCHEDULED
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration
>>>>> (Unnamed
>>>>> >> >>> >>>>> >> Delta
>>>>> >> >>> >>>>> >> Iteration)) (1/1) switched to DEPLOYING
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>>>>> >> >>> >>>>> >> (/tmp/tmpBhOsLd) -
>>>>> >> >>> >>>>> >> UTF-8) -> Map
>>>>> (org.apache.flink.api.scala.DataSet$$anon$1)
>>>>> >> >>> >>>>> >> (1/1)
>>>>> >> >>> >>>>> >> switched to
>>>>> >> >>> >>>>> >> RUNNING
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to
>>>>> >> >>> >>>>> >> RUNNING
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration
>>>>> (Unnamed
>>>>> >> >>> >>>>> >> Delta
>>>>> >> >>> >>>>> >> Iteration)) (1/1) switched to RUNNING
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>>>> >> >>> >>>>> >> ->
>>>>> >> >>> >>>>> >> Map
>>>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>>>> >> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1)
>>>>> switched to
>>>>> >> >>> >>>>> >> SCHEDULED
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>>>> >> >>> >>>>> >> ->
>>>>> >> >>> >>>>> >> Map
>>>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>>>> >> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1)
>>>>> switched to
>>>>> >> >>> >>>>> >> DEPLOYING
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>>>> >> >>> >>>>> >> ->
>>>>> >> >>> >>>>> >> Map
>>>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>>>> switched
>>>>> >> >>> >>>>> >> to
>>>>> >> >>> >>>>> >> SCHEDULED
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>>>> >> >>> >>>>> >> ->
>>>>> >> >>> >>>>> >> Map
>>>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>>>> switched
>>>>> >> >>> >>>>> >> to
>>>>> >> >>> >>>>> >> DEPLOYING
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>>>> >> >>> >>>>> >> ->
>>>>> >> >>> >>>>> >> Map
>>>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>>>> >> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1)
>>>>> switched to
>>>>> >> >>> >>>>> >> RUNNING
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>>>> >> >>> >>>>> >> ->
>>>>> >> >>> >>>>> >> Map
>>>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>>>> switched
>>>>> >> >>> >>>>> >> to
>>>>> >> >>> >>>>> >> RUNNING
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>>>> >> >>> >>>>> >> ->
>>>>> >> >>> >>>>> >> Map
>>>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>>>> >> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1)
>>>>> switched to
>>>>> >> >>> >>>>> >> FAILED
>>>>> >> >>> >>>>> >> java.lang.IllegalStateException: This stub is not part
>>>>> of an
>>>>> >> >>> >>>>> >> iteration
>>>>> >> >>> >>>>> >> step function.
>>>>> >> >>> >>>>> >> at
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
>>>>> >> >>> >>>>> >> at
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
>>>>> >> >>> >>>>> >> at
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
>>>>> >> >>> >>>>> >> at
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
>>>>> >> >>> >>>>> >> at
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
>>>>> >> >>> >>>>> >> at
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
>>>>> >> >>> >>>>> >> at
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
>>>>> >> >>> >>>>> >> at
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
>>>>> >> >>> >>>>> >> at
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
>>>>> >> >>> >>>>> >> at
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
>>>>> >> >>> >>>>> >> at
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
>>>>> >> >>> >>>>> >> at
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
>>>>> >> >>> >>>>> >> at java.lang.Thread.run(Thread.java:745)
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: Job execution switched to status
>>>>> FAILING
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to
>>>>> >> >>> >>>>> >> CANCELING
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
>>>>> >> >>> >>>>> >> (/tmp/tmpBhOsLd) -
>>>>> >> >>> >>>>> >> UTF-8) -> Map
>>>>> (org.apache.flink.api.scala.DataSet$$anon$1)
>>>>> >> >>> >>>>> >> (1/1)
>>>>> >> >>> >>>>> >> switched to
>>>>> >> >>> >>>>> >> CANCELING
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1)
>>>>> switched to
>>>>> >> >>> >>>>> >> CANCELING
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration
>>>>> (Unnamed
>>>>> >> >>> >>>>> >> Delta
>>>>> >> >>> >>>>> >> Iteration)) (1/1) switched to CANCELING
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: Map
>>>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1)
>>>>> >> >>> >>>>> >> (1/1) switched to CANCELED
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: DataSink(TextOutputFormat
>>>>> >> >>> >>>>> >> (/tmp/tmplSYJ7S)
>>>>> >> >>> >>>>> >> -
>>>>> >> >>> >>>>> >> UTF-8)
>>>>> >> >>> >>>>> >> (1/1) switched to CANCELED
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: Sync (WorksetIteration (Unnamed
>>>>> Delta
>>>>> >> >>> >>>>> >> Iteration))
>>>>> >> >>> >>>>> >> (1/1) switched to CANCELED
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>>>> >> >>> >>>>> >> ->
>>>>> >> >>> >>>>> >> Map
>>>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>>>> switched
>>>>> >> >>> >>>>> >> to
>>>>> >> >>> >>>>> >> CANCELING
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: Map
>>>>> >> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$2)
>>>>> >> >>> >>>>> >> (1/1)
>>>>> >> >>> >>>>> >> switched to CANCELED
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: SolutionSet Delta (1/1) switched
>>>>> to
>>>>> >> >>> >>>>> >> CANCELED
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to
>>>>> >> >>> >>>>> >> CANCELED
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
>>>>> >> >>> >>>>> >> (/tmp/tmpBhOsLd) -
>>>>> >> >>> >>>>> >> UTF-8) -> Map
>>>>> (org.apache.flink.api.scala.DataSet$$anon$1)
>>>>> >> >>> >>>>> >> (1/1)
>>>>> >> >>> >>>>> >> switched to
>>>>> >> >>> >>>>> >> CANCELED
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1)
>>>>> switched to
>>>>> >> >>> >>>>> >> CANCELED
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>>>> >> >>> >>>>> >> ->
>>>>> >> >>> >>>>> >> Map
>>>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>>>> switched
>>>>> >> >>> >>>>> >> to
>>>>> >> >>> >>>>> >> CANCELED
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration
>>>>> (Unnamed
>>>>> >> >>> >>>>> >> Delta
>>>>> >> >>> >>>>> >> Iteration)) (1/1) switched to CANCELED
>>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: Job execution switched to status
>>>>> FAILED
>>>>> >> >>> >>>>> >> Error: The program execution failed:
>>>>> >> >>> >>>>> >> java.lang.IllegalStateException: This
>>>>> >> >>> >>>>> >> stub is not part of an iteration step function.
>>>>> >> >>> >>>>> >> at
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
>>>>> >> >>> >>>>> >> at
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
>>>>> >> >>> >>>>> >> at
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
>>>>> >> >>> >>>>> >> at
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
>>>>> >> >>> >>>>> >> at
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
>>>>> >> >>> >>>>> >> at
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
>>>>> >> >>> >>>>> >> at
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
>>>>> >> >>> >>>>> >> at
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
>>>>> >> >>> >>>>> >> at
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
>>>>> >> >>> >>>>> >> at
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
>>>>> >> >>> >>>>> >> at
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
>>>>> >> >>> >>>>> >> at
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >>
>>>>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
>>>>> >> >>> >>>>> >> at java.lang.Thread.run(Thread.java:745)
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >> On Tue, Oct 14, 2014 at 3:32 PM, Maximilian Alber
>>>>> >> >>> >>>>> >> <al...@gmail.com> wrote:
>>>>> >> >>> >>>>> >>>
>>>>> >> >>> >>>>> >>> Should work now.
>>>>> >> >>> >>>>> >>> Cheers
>>>>> >> >>> >>>>> >>>
>>>>> >> >>> >>>>> >>> On Fri, Oct 10, 2014 at 3:38 PM, Maximilian Alber
>>>>> >> >>> >>>>> >>> <al...@gmail.com> wrote:
>>>>> >> >>> >>>>> >>>>
>>>>> >> >>> >>>>> >>>> Ok, thanks.
>>>>> >> >>> >>>>> >>>> Please let me know when it is fixed.
>>>>> >> >>> >>>>> >>>>
>>>>> >> >>> >>>>> >>>> Cheers
>>>>> >> >>> >>>>> >>>> Max
>>>>> >> >>> >>>>> >>>>
>>>>> >> >>> >>>>> >>>> On Fri, Oct 10, 2014 at 1:34 PM, Stephan Ewen
>>>>> >> >>> >>>>> >>>> <se...@apache.org>
>>>>> >> >>> >>>>> >>>> wrote:
>>>>> >> >>> >>>>> >>>>>
>>>>> >> >>> >>>>> >>>>> Thank you, I will look into that...
>>>>> >> >>> >>>>> >>>>
>>>>> >> >>> >>>>> >>>>
>>>>> >> >>> >>>>> >>>
>>>>> >> >>> >>>>> >>
>>>>> >> >>> >>>>> >
>>>>> >> >>> >>>>
>>>>> >> >>> >>>>
>>>>> >> >>> >>>
>>>>> >> >>> >>
>>>>> >> >>> >
>>>>> >> >>
>>>>> >> >>
>>>>> >> >
>>>>> >
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Forced to use Solution Set in Step Function
Posted by Maximilian Alber <al...@gmail.com>.
Hi Stephan!
It worked with the development snapshot. I never tried it with the new
release. If you want me to try it out, just say so.
Cheers,
Max
On Mon, Nov 10, 2014 at 5:13 PM, Stephan Ewen <se...@apache.org> wrote:
> Hi!
>
> The 0.7.0 release should have fixed that problem. Have you had a chance to
> try that out?
>
> Greetings,
> Stephan
>
>
> On Wed, Oct 15, 2014 at 3:43 PM, Maximilian Alber <
> alber.maximilian@gmail.com> wrote:
>
>> @Aljoscha Sorry, I just tried my workaround. There are some minor
>> conceptual bugs (caused by the id's...). I attached the new version.
>> Unfortunately there the compiler breaks. An issue is already open.
>>
>> On Wed, Oct 15, 2014 at 11:53 AM, Maximilian Alber <
>> alber.maximilian@gmail.com> wrote:
>>
>>> Ok.
>>>
>>> Here is a input variant:
>>> flink run -v bump_boost-0.1.jar -c bumpboost.Job x_file=X y_file=Y
>>> out_file=/tmp/tmpnWYamw random_file=random_file dimensions=1 N=100
>>> width_candidates_file=/tmp/tmpTJ4LDh iterations=30 multi_bump_boost=0
>>> gradient_descent_iterations=30 cache=False min_width=-4 max_width=6
>>> min_width_update=1e-08 max_width_update=10
>>>
>>> width_candidates_file is not needed by now. X, Y and random_file are
>>> attached.
>>>
>>> If you have problems, running it, let me know!
>>> Thanks!
>>>
>>> On Tue, Oct 14, 2014 at 10:40 PM, Aljoscha Krettek <al...@apache.org>
>>> wrote:
>>>
>>>> Transferring to variables: Unfortunately not possible right now but we
>>>> are working on it.
>>>>
>>>> On Tue, Oct 14, 2014 at 8:53 PM, Maximilian Alber
>>>> <al...@gmail.com> wrote:
>>>> > @ Stephan: Thanks! So I gonna switch!
>>>> >
>>>> > Sorry, my bad. I will provide you some sample by tomorrow morning.
>>>> >
>>>> > Yes. Workaround, because I cannot transfer them into variables, can I
>>>> by now
>>>> > (or will I ever)?
>>>> >
>>>> > Maybe some explanation to my solution:
>>>> > - X is for my a matrix of shape (N, d). Modeled in Flink as dataset of
>>>> > vectors. Each Vector has an ID which is the row number and an array
>>>> with
>>>> > numbers, the actual row.
>>>> > - Y is for my a matrix of shape (N, 1) thus actually a column-vector.
>>>> > - old_sum is either a scalar if d == 1 or a row-vector aka matrix of
>>>> shape
>>>> > (1, N) or a Dataset with one Vector. (By now I have the convention to
>>>> give
>>>> > id -1 to them, comes from a former workaround...)
>>>> >
>>>> > The whole ID story comes from the fact that I need to know which stuff
>>>> > belongs together in mathematical operations (see my zip function).
>>>> You can
>>>> > look that up in util.scala, that's kind of my math library. I don't
>>>> want to
>>>> > imagine the mess in Java :)
>>>> >
>>>> > Cheers
>>>> > Max
>>>> >
>>>> >
>>>> >
>>>> > On Tue, Oct 14, 2014 at 6:28 PM, Aljoscha Krettek <
>>>> aljoscha@apache.org>
>>>> > wrote:
>>>> >>
>>>> >> Could you maybe also give some examples for the input expected by
>>>> your
>>>> >> program?
>>>> >>
>>>> >> Also, the residual DataSet contains several Vectors while the sum (or
>>>> >> old_sum) DataSet is always only contains 1 Vector. Correct?
>>>> >>
>>>> >> On Tue, Oct 14, 2014 at 6:04 PM, Stephan Ewen <se...@apache.org>
>>>> wrote:
>>>> >> > BTW: The current master allows you to not join with the solution
>>>> set,
>>>> >> > and
>>>> >> > only use it to accumulate data.
>>>> >> >
>>>> >> > On Tue, Oct 14, 2014 at 5:29 PM, Maximilian Alber
>>>> >> > <al...@gmail.com> wrote:
>>>> >> >>
>>>> >> >> Ok, that's possible too.
>>>> >> >>
>>>> >> >> VectorDataSet is just scala magic to ease my life (See below). If
>>>> you
>>>> >> >> want
>>>> >> >> to take a look, I appended the package. The main code is in
>>>> >> >> BumpBoost.scala.
>>>> >> >> In util.scala is the vector stuff.
>>>> >> >> Thanks!
>>>> >> >>
>>>> >> >> class VectorDataSet(X: DataSet[Vector]){
>>>> >> >> def + (Y: DataSet[Vector]) = VectorDataSet.add(X, Y)
>>>> >> >> def - (Y: DataSet[Vector]) = VectorDataSet.subt(X, Y)
>>>> >> >> def * (Y: DataSet[Vector]) = VectorDataSet.mult(X, Y)
>>>> >> >> def / (Y: DataSet[Vector]) = VectorDataSet.div(X, Y)
>>>> >> >>
>>>> >> >> def zip(Y: DataSet[Vector]) = VectorDataSet.zip(X, Y)
>>>> >> >> def dot(Y: DataSet[Vector]) = VectorDataSet.dot(X, Y)
>>>> >> >> def sumV() = VectorDataSet.sumV(X)
>>>> >> >> }
>>>> >> >> object VectorDataSet {
>>>> >> >> def add(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map
>>>> {x =>
>>>> >> >> x._1 + x._2}
>>>> >> >> def subt(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2
>>>> map {x
>>>> >> >> =>
>>>> >> >> x._1 - x._2}
>>>> >> >> def mult(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2
>>>> map {x
>>>> >> >> =>
>>>> >> >> x._1 * x._2}
>>>> >> >> def div(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map
>>>> {x =>
>>>> >> >> x._1 / x._2}
>>>> >> >>
>>>> >> >> def zip(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 join X2
>>>> where
>>>> >> >> "id"
>>>> >> >> equalTo "id"
>>>> >> >> def dot(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 * X2 sumV
>>>> >> >> def sumV(X1: DataSet[Vector]) = X1 reduce {_ + _}
>>>> >> >>
>>>> >> >> implicit def vectorDataSetWrapper(ds: DataSet[Vector]) = new
>>>> >> >> VectorDataSet(ds)
>>>> >> >> }
>>>> >> >>
>>>> >> >>
>>>> >> >> On Tue, Oct 14, 2014 at 5:21 PM, Aljoscha Krettek <
>>>> aljoscha@apache.org>
>>>> >> >> wrote:
>>>> >> >>>
>>>> >> >>> Maybe you could use the residual_2 data set as a broadcast
>>>> dataset.
>>>> >> >>> i.e. make in available in the operation that adds the residual
>>>> for the
>>>> >> >>> current iteration number to the old_sum. (I'm not sure what the
>>>> >> >>> VectorDataSet.add() method does here). If you gave me the
>>>> complete
>>>> >> >>> code I could try finding an elegant solution to that problem.
>>>> >> >>>
>>>> >> >>> On Tue, Oct 14, 2014 at 5:15 PM, Stephan Ewen <se...@apache.org>
>>>> >> >>> wrote:
>>>> >> >>> > That is an interesting case. Everything that is loop invariant
>>>> is
>>>> >> >>> > computed
>>>> >> >>> > once outside the loop. You are looking for a way to make this
>>>> part
>>>> >> >>> > of
>>>> >> >>> > the
>>>> >> >>> > loop.
>>>> >> >>> >
>>>> >> >>> > Can you try making the filter part of the
>>>> >> >>> > "VectorDataSet.add(old_sum,
>>>> >> >>> > current)" operation?
>>>> >> >>> >
>>>> >> >>> > On Tue, Oct 14, 2014 at 5:05 PM, Maximilian Alber
>>>> >> >>> > <al...@gmail.com> wrote:
>>>> >> >>> >>
>>>> >> >>> >> The deltaiteration calculates the cumulative sum (prefix sum)
>>>> of
>>>> >> >>> >> residual_2.
>>>> >> >>> >>
>>>> >> >>> >> I'm not sure if got you idea. I could add residual_2 to the
>>>> >> >>> >> workset.
>>>> >> >>> >> But
>>>> >> >>> >> in that case I need to separate the "old_sum"(the sum up to
>>>> now)
>>>> >> >>> >> and
>>>> >> >>> >> residual_2 each iteration. Actually I had that before, then I
>>>> tried
>>>> >> >>> >> the
>>>> >> >>> >> Scala closure to clean up the code.
>>>> >> >>> >>
>>>> >> >>> >> On Tue, Oct 14, 2014 at 4:57 PM, Fabian Hueske <
>>>> fhueske@apache.org>
>>>> >> >>> >> wrote:
>>>> >> >>> >>>
>>>> >> >>> >>> Jep, I see you point.
>>>> >> >>> >>> Conceptually, all data that changes and affects the result
>>>> of an
>>>> >> >>> >>> iteration should be part of the workset.
>>>> >> >>> >>> Hence, the model kind of assumes that the datum
>>>> "superStepNumber"
>>>> >> >>> >>> should
>>>> >> >>> >>> be part of the workset.
>>>> >> >>> >>>
>>>> >> >>> >>> I am not familiar with your application, but would it make
>>>> sense
>>>> >> >>> >>> to
>>>> >> >>> >>> add
>>>> >> >>> >>> the number as an additional attribute to the workset data
>>>> set and
>>>> >> >>> >>> increase
>>>> >> >>> >>> it manually?
>>>> >> >>> >>>
>>>> >> >>> >>> 2014-10-14 16:45 GMT+02:00 Maximilian Alber
>>>> >> >>> >>> <al...@gmail.com>:
>>>> >> >>> >>>>
>>>> >> >>> >>>> Ok, sounds true, but somehow I would like to execute it
>>>> inside of
>>>> >> >>> >>>> it. So
>>>> >> >>> >>>> I probably need to do some nonsense work to make it part of
>>>> it?
>>>> >> >>> >>>>
>>>> >> >>> >>>> On Tue, Oct 14, 2014 at 4:36 PM, Aljoscha Krettek
>>>> >> >>> >>>> <al...@apache.org>
>>>> >> >>> >>>> wrote:
>>>> >> >>> >>>>>
>>>> >> >>> >>>>> Dammit you beat me to it. But yes, this is exactly what I
>>>> was
>>>> >> >>> >>>>> just
>>>> >> >>> >>>>> writing.
>>>> >> >>> >>>>>
>>>> >> >>> >>>>> On Tue, Oct 14, 2014 at 4:35 PM, Fabian Hueske
>>>> >> >>> >>>>> <fh...@apache.org>
>>>> >> >>> >>>>> wrote:
>>>> >> >>> >>>>> > Hi,
>>>> >> >>> >>>>> >
>>>> >> >>> >>>>> > I'm not super familiar with the iterations, but my guess
>>>> would
>>>> >> >>> >>>>> > be
>>>> >> >>> >>>>> > that the
>>>> >> >>> >>>>> > filter is not evaluated as part of the iteration.
>>>> >> >>> >>>>> > Since it is not connect to the workset, the filter is
>>>> not part
>>>> >> >>> >>>>> > of
>>>> >> >>> >>>>> > the
>>>> >> >>> >>>>> > loop
>>>> >> >>> >>>>> > and evaluated once outside where no superset number is
>>>> >> >>> >>>>> > available.
>>>> >> >>> >>>>> > I guess, moving the filter outside of the loop gives the
>>>> same
>>>> >> >>> >>>>> > error.
>>>> >> >>> >>>>> >
>>>> >> >>> >>>>> > Cheers, Fabian
>>>> >> >>> >>>>> >
>>>> >> >>> >>>>> >
>>>> >> >>> >>>>> >
>>>> >> >>> >>>>> > 2014-10-14 16:18 GMT+02:00 Maximilian Alber
>>>> >> >>> >>>>> > <al...@gmail.com>:
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >> Hmm or maybe not. With this code I get some strange
>>>> error:
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >> def createPlan_find_center(env: ExecutionEnvironment) =
>>>> {
>>>> >> >>> >>>>> >> val X = env readTextFile config.xFile map
>>>> >> >>> >>>>> >> {Vector.parseFromString(config.dimensions, _)};
>>>> >> >>> >>>>> >> val residual = env readTextFile config.yFile map
>>>> >> >>> >>>>> >> {Vector.parseFromString(_)};
>>>> >> >>> >>>>> >> val randoms = env readTextFile config.randomFile map
>>>> >> >>> >>>>> >> {Vector.parseFromString(_)}
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >> val residual_2 = residual * residual
>>>> >> >>> >>>>> >> val ys = (residual_2 sumV) * (randoms filter {_.id ==
>>>> 0})
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >> val emptyDataSet = env.fromCollection[Vector](Seq())
>>>> >> >>> >>>>> >> val sumVector =
>>>> >> >>> >>>>> >> env.fromCollection(Seq(Vector.zeros(config.dimensions)))
>>>> >> >>> >>>>> >> val cumSum = emptyDataSet.iterateDelta(sumVector,
>>>> config.N,
>>>> >> >>> >>>>> >> Array("id")) {
>>>> >> >>> >>>>> >> (solutionset, old_sum) =>
>>>> >> >>> >>>>> >> val current = residual_2 filter (new
>>>> >> >>> >>>>> >> RichFilterFunction[Vector]{
>>>> >> >>> >>>>> >> def filter(x: Vector) = x.id ==
>>>> >> >>> >>>>> >> (getIterationRuntimeContext.getSuperstepNumber)
>>>> >> >>> >>>>> >> })
>>>> >> >>> >>>>> >> val sum = VectorDataSet.add(old_sum, current)
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >> (sum map (new RichMapFunction[Vector, Vector]{
>>>> >> >>> >>>>> >> def map(x: Vector) = new
>>>> >> >>> >>>>> >> Vector(getIterationRuntimeContext.getSuperstepNumber,
>>>> >> >>> >>>>> >> x.values)
>>>> >> >>> >>>>> >> }),
>>>> >> >>> >>>>> >> sum)
>>>> >> >>> >>>>> >> }
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >> Error:
>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: Job execution switched to status
>>>> RUNNING
>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1)
>>>> switched to
>>>> >> >>> >>>>> >> SCHEDULED
>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1)
>>>> switched to
>>>> >> >>> >>>>> >> DEPLOYING
>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to
>>>> >> >>> >>>>> >> SCHEDULED
>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to
>>>> >> >>> >>>>> >> DEPLOYING
>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>>>> >> >>> >>>>> >> (/tmp/tmpBhOsLd) -
>>>> >> >>> >>>>> >> UTF-8) -> Map
>>>> (org.apache.flink.api.scala.DataSet$$anon$1)
>>>> >> >>> >>>>> >> (1/1)
>>>> >> >>> >>>>> >> switched to
>>>> >> >>> >>>>> >> SCHEDULED
>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>>>> >> >>> >>>>> >> (/tmp/tmpBhOsLd) -
>>>> >> >>> >>>>> >> UTF-8) -> Map
>>>> (org.apache.flink.api.scala.DataSet$$anon$1)
>>>> >> >>> >>>>> >> (1/1)
>>>> >> >>> >>>>> >> switched to
>>>> >> >>> >>>>> >> DEPLOYING
>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1)
>>>> switched to
>>>> >> >>> >>>>> >> RUNNING
>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration
>>>> (Unnamed
>>>> >> >>> >>>>> >> Delta
>>>> >> >>> >>>>> >> Iteration)) (1/1) switched to SCHEDULED
>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration
>>>> (Unnamed
>>>> >> >>> >>>>> >> Delta
>>>> >> >>> >>>>> >> Iteration)) (1/1) switched to DEPLOYING
>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>>>> >> >>> >>>>> >> (/tmp/tmpBhOsLd) -
>>>> >> >>> >>>>> >> UTF-8) -> Map
>>>> (org.apache.flink.api.scala.DataSet$$anon$1)
>>>> >> >>> >>>>> >> (1/1)
>>>> >> >>> >>>>> >> switched to
>>>> >> >>> >>>>> >> RUNNING
>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to
>>>> >> >>> >>>>> >> RUNNING
>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration
>>>> (Unnamed
>>>> >> >>> >>>>> >> Delta
>>>> >> >>> >>>>> >> Iteration)) (1/1) switched to RUNNING
>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>>> >> >>> >>>>> >> ->
>>>> >> >>> >>>>> >> Map
>>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>>> >> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched
>>>> to
>>>> >> >>> >>>>> >> SCHEDULED
>>>> >> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>>> >> >>> >>>>> >> ->
>>>> >> >>> >>>>> >> Map
>>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>>> >> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched
>>>> to
>>>> >> >>> >>>>> >> DEPLOYING
>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>>> >> >>> >>>>> >> ->
>>>> >> >>> >>>>> >> Map
>>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>>> switched
>>>> >> >>> >>>>> >> to
>>>> >> >>> >>>>> >> SCHEDULED
>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>>> >> >>> >>>>> >> ->
>>>> >> >>> >>>>> >> Map
>>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>>> switched
>>>> >> >>> >>>>> >> to
>>>> >> >>> >>>>> >> DEPLOYING
>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>>> >> >>> >>>>> >> ->
>>>> >> >>> >>>>> >> Map
>>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>>> >> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched
>>>> to
>>>> >> >>> >>>>> >> RUNNING
>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>>> >> >>> >>>>> >> ->
>>>> >> >>> >>>>> >> Map
>>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>>> switched
>>>> >> >>> >>>>> >> to
>>>> >> >>> >>>>> >> RUNNING
>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>>> >> >>> >>>>> >> ->
>>>> >> >>> >>>>> >> Map
>>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>>> >> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched
>>>> to
>>>> >> >>> >>>>> >> FAILED
>>>> >> >>> >>>>> >> java.lang.IllegalStateException: This stub is not part
>>>> of an
>>>> >> >>> >>>>> >> iteration
>>>> >> >>> >>>>> >> step function.
>>>> >> >>> >>>>> >> at
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
>>>> >> >>> >>>>> >> at
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
>>>> >> >>> >>>>> >> at
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
>>>> >> >>> >>>>> >> at
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
>>>> >> >>> >>>>> >> at
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
>>>> >> >>> >>>>> >> at
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
>>>> >> >>> >>>>> >> at
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
>>>> >> >>> >>>>> >> at
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
>>>> >> >>> >>>>> >> at
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
>>>> >> >>> >>>>> >> at
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
>>>> >> >>> >>>>> >> at
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
>>>> >> >>> >>>>> >> at
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
>>>> >> >>> >>>>> >> at java.lang.Thread.run(Thread.java:745)
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: Job execution switched to status
>>>> FAILING
>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to
>>>> >> >>> >>>>> >> CANCELING
>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
>>>> >> >>> >>>>> >> (/tmp/tmpBhOsLd) -
>>>> >> >>> >>>>> >> UTF-8) -> Map
>>>> (org.apache.flink.api.scala.DataSet$$anon$1)
>>>> >> >>> >>>>> >> (1/1)
>>>> >> >>> >>>>> >> switched to
>>>> >> >>> >>>>> >> CANCELING
>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1)
>>>> switched to
>>>> >> >>> >>>>> >> CANCELING
>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration
>>>> (Unnamed
>>>> >> >>> >>>>> >> Delta
>>>> >> >>> >>>>> >> Iteration)) (1/1) switched to CANCELING
>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: Map
>>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1)
>>>> >> >>> >>>>> >> (1/1) switched to CANCELED
>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: DataSink(TextOutputFormat
>>>> >> >>> >>>>> >> (/tmp/tmplSYJ7S)
>>>> >> >>> >>>>> >> -
>>>> >> >>> >>>>> >> UTF-8)
>>>> >> >>> >>>>> >> (1/1) switched to CANCELED
>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: Sync (WorksetIteration (Unnamed
>>>> Delta
>>>> >> >>> >>>>> >> Iteration))
>>>> >> >>> >>>>> >> (1/1) switched to CANCELED
>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>>> >> >>> >>>>> >> ->
>>>> >> >>> >>>>> >> Map
>>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>>> switched
>>>> >> >>> >>>>> >> to
>>>> >> >>> >>>>> >> CANCELING
>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: Map
>>>> >> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$2)
>>>> >> >>> >>>>> >> (1/1)
>>>> >> >>> >>>>> >> switched to CANCELED
>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: SolutionSet Delta (1/1) switched to
>>>> >> >>> >>>>> >> CANCELED
>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to
>>>> >> >>> >>>>> >> CANCELED
>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
>>>> >> >>> >>>>> >> (/tmp/tmpBhOsLd) -
>>>> >> >>> >>>>> >> UTF-8) -> Map
>>>> (org.apache.flink.api.scala.DataSet$$anon$1)
>>>> >> >>> >>>>> >> (1/1)
>>>> >> >>> >>>>> >> switched to
>>>> >> >>> >>>>> >> CANCELED
>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1)
>>>> switched to
>>>> >> >>> >>>>> >> CANCELED
>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>>> >> >>> >>>>> >> ->
>>>> >> >>> >>>>> >> Map
>>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>>> switched
>>>> >> >>> >>>>> >> to
>>>> >> >>> >>>>> >> CANCELED
>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration
>>>> (Unnamed
>>>> >> >>> >>>>> >> Delta
>>>> >> >>> >>>>> >> Iteration)) (1/1) switched to CANCELED
>>>> >> >>> >>>>> >> 10/14/2014 15:57:36: Job execution switched to status
>>>> FAILED
>>>> >> >>> >>>>> >> Error: The program execution failed:
>>>> >> >>> >>>>> >> java.lang.IllegalStateException: This
>>>> >> >>> >>>>> >> stub is not part of an iteration step function.
>>>> >> >>> >>>>> >> at
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
>>>> >> >>> >>>>> >> at
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
>>>> >> >>> >>>>> >> at
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
>>>> >> >>> >>>>> >> at
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
>>>> >> >>> >>>>> >> at
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
>>>> >> >>> >>>>> >> at
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
>>>> >> >>> >>>>> >> at
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
>>>> >> >>> >>>>> >> at
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
>>>> >> >>> >>>>> >> at
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
>>>> >> >>> >>>>> >> at
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
>>>> >> >>> >>>>> >> at
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
>>>> >> >>> >>>>> >> at
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >>
>>>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
>>>> >> >>> >>>>> >> at java.lang.Thread.run(Thread.java:745)
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >> On Tue, Oct 14, 2014 at 3:32 PM, Maximilian Alber
>>>> >> >>> >>>>> >> <al...@gmail.com> wrote:
>>>> >> >>> >>>>> >>>
>>>> >> >>> >>>>> >>> Should work now.
>>>> >> >>> >>>>> >>> Cheers
>>>> >> >>> >>>>> >>>
>>>> >> >>> >>>>> >>> On Fri, Oct 10, 2014 at 3:38 PM, Maximilian Alber
>>>> >> >>> >>>>> >>> <al...@gmail.com> wrote:
>>>> >> >>> >>>>> >>>>
>>>> >> >>> >>>>> >>>> Ok, thanks.
>>>> >> >>> >>>>> >>>> Please let me know when it is fixed.
>>>> >> >>> >>>>> >>>>
>>>> >> >>> >>>>> >>>> Cheers
>>>> >> >>> >>>>> >>>> Max
>>>> >> >>> >>>>> >>>>
>>>> >> >>> >>>>> >>>> On Fri, Oct 10, 2014 at 1:34 PM, Stephan Ewen
>>>> >> >>> >>>>> >>>> <se...@apache.org>
>>>> >> >>> >>>>> >>>> wrote:
>>>> >> >>> >>>>> >>>>>
>>>> >> >>> >>>>> >>>>> Thank you, I will look into that...
>>>> >> >>> >>>>> >>>>
>>>> >> >>> >>>>> >>>>
>>>> >> >>> >>>>> >>>
>>>> >> >>> >>>>> >>
>>>> >> >>> >>>>> >
>>>> >> >>> >>>>
>>>> >> >>> >>>>
>>>> >> >>> >>>
>>>> >> >>> >>
>>>> >> >>> >
>>>> >> >>
>>>> >> >>
>>>> >> >
>>>> >
>>>> >
>>>>
>>>
>>>
>>
>
Re: Forced to use Solution Set in Step Function
Posted by Stephan Ewen <se...@apache.org>.
Hi!
The 0.7.0 release should have fixed that problem. Have you had a chance to
try that out?
Greetings,
Stephan
On Wed, Oct 15, 2014 at 3:43 PM, Maximilian Alber <
alber.maximilian@gmail.com> wrote:
> @Aljoscha Sorry, I just tried my workaround. There are some minor
> conceptual bugs (caused by the id's...). I attached the new version.
> Unfortunately there the compiler breaks. An issue is already open.
>
> On Wed, Oct 15, 2014 at 11:53 AM, Maximilian Alber <
> alber.maximilian@gmail.com> wrote:
>
>> Ok.
>>
>> Here is a input variant:
>> flink run -v bump_boost-0.1.jar -c bumpboost.Job x_file=X y_file=Y
>> out_file=/tmp/tmpnWYamw random_file=random_file dimensions=1 N=100
>> width_candidates_file=/tmp/tmpTJ4LDh iterations=30 multi_bump_boost=0
>> gradient_descent_iterations=30 cache=False min_width=-4 max_width=6
>> min_width_update=1e-08 max_width_update=10
>>
>> width_candidates_file is not needed by now. X, Y and random_file are
>> attached.
>>
>> If you have problems, running it, let me know!
>> Thanks!
>>
>> On Tue, Oct 14, 2014 at 10:40 PM, Aljoscha Krettek <al...@apache.org>
>> wrote:
>>
>>> Transferring to variables: Unfortunately not possible right now but we
>>> are working on it.
>>>
>>> On Tue, Oct 14, 2014 at 8:53 PM, Maximilian Alber
>>> <al...@gmail.com> wrote:
>>> > @ Stephan: Thanks! So I gonna switch!
>>> >
>>> > Sorry, my bad. I will provide you some sample by tomorrow morning.
>>> >
>>> > Yes. Workaround, because I cannot transfer them into variables, can I
>>> by now
>>> > (or will I ever)?
>>> >
>>> > Maybe some explanation to my solution:
>>> > - X is for my a matrix of shape (N, d). Modeled in Flink as dataset of
>>> > vectors. Each Vector has an ID which is the row number and an array
>>> with
>>> > numbers, the actual row.
>>> > - Y is for my a matrix of shape (N, 1) thus actually a column-vector.
>>> > - old_sum is either a scalar if d == 1 or a row-vector aka matrix of
>>> shape
>>> > (1, N) or a Dataset with one Vector. (By now I have the convention to
>>> give
>>> > id -1 to them, comes from a former workaround...)
>>> >
>>> > The whole ID story comes from the fact that I need to know which stuff
>>> > belongs together in mathematical operations (see my zip function). You
>>> can
>>> > look that up in util.scala, that's kind of my math library. I don't
>>> want to
>>> > imagine the mess in Java :)
>>> >
>>> > Cheers
>>> > Max
>>> >
>>> >
>>> >
>>> > On Tue, Oct 14, 2014 at 6:28 PM, Aljoscha Krettek <aljoscha@apache.org
>>> >
>>> > wrote:
>>> >>
>>> >> Could you maybe also give some examples for the input expected by your
>>> >> program?
>>> >>
>>> >> Also, the residual DataSet contains several Vectors while the sum (or
>>> >> old_sum) DataSet is always only contains 1 Vector. Correct?
>>> >>
>>> >> On Tue, Oct 14, 2014 at 6:04 PM, Stephan Ewen <se...@apache.org>
>>> wrote:
>>> >> > BTW: The current master allows you to not join with the solution
>>> set,
>>> >> > and
>>> >> > only use it to accumulate data.
>>> >> >
>>> >> > On Tue, Oct 14, 2014 at 5:29 PM, Maximilian Alber
>>> >> > <al...@gmail.com> wrote:
>>> >> >>
>>> >> >> Ok, that's possible too.
>>> >> >>
>>> >> >> VectorDataSet is just scala magic to ease my life (See below). If
>>> you
>>> >> >> want
>>> >> >> to take a look, I appended the package. The main code is in
>>> >> >> BumpBoost.scala.
>>> >> >> In util.scala is the vector stuff.
>>> >> >> Thanks!
>>> >> >>
>>> >> >> class VectorDataSet(X: DataSet[Vector]){
>>> >> >> def + (Y: DataSet[Vector]) = VectorDataSet.add(X, Y)
>>> >> >> def - (Y: DataSet[Vector]) = VectorDataSet.subt(X, Y)
>>> >> >> def * (Y: DataSet[Vector]) = VectorDataSet.mult(X, Y)
>>> >> >> def / (Y: DataSet[Vector]) = VectorDataSet.div(X, Y)
>>> >> >>
>>> >> >> def zip(Y: DataSet[Vector]) = VectorDataSet.zip(X, Y)
>>> >> >> def dot(Y: DataSet[Vector]) = VectorDataSet.dot(X, Y)
>>> >> >> def sumV() = VectorDataSet.sumV(X)
>>> >> >> }
>>> >> >> object VectorDataSet {
>>> >> >> def add(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map
>>> {x =>
>>> >> >> x._1 + x._2}
>>> >> >> def subt(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map
>>> {x
>>> >> >> =>
>>> >> >> x._1 - x._2}
>>> >> >> def mult(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map
>>> {x
>>> >> >> =>
>>> >> >> x._1 * x._2}
>>> >> >> def div(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map
>>> {x =>
>>> >> >> x._1 / x._2}
>>> >> >>
>>> >> >> def zip(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 join X2
>>> where
>>> >> >> "id"
>>> >> >> equalTo "id"
>>> >> >> def dot(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 * X2 sumV
>>> >> >> def sumV(X1: DataSet[Vector]) = X1 reduce {_ + _}
>>> >> >>
>>> >> >> implicit def vectorDataSetWrapper(ds: DataSet[Vector]) = new
>>> >> >> VectorDataSet(ds)
>>> >> >> }
>>> >> >>
>>> >> >>
>>> >> >> On Tue, Oct 14, 2014 at 5:21 PM, Aljoscha Krettek <
>>> aljoscha@apache.org>
>>> >> >> wrote:
>>> >> >>>
>>> >> >>> Maybe you could use the residual_2 data set as a broadcast
>>> dataset.
>>> >> >>> i.e. make in available in the operation that adds the residual
>>> for the
>>> >> >>> current iteration number to the old_sum. (I'm not sure what the
>>> >> >>> VectorDataSet.add() method does here). If you gave me the complete
>>> >> >>> code I could try finding an elegant solution to that problem.
>>> >> >>>
>>> >> >>> On Tue, Oct 14, 2014 at 5:15 PM, Stephan Ewen <se...@apache.org>
>>> >> >>> wrote:
>>> >> >>> > That is an interesting case. Everything that is loop invariant
>>> is
>>> >> >>> > computed
>>> >> >>> > once outside the loop. You are looking for a way to make this
>>> part
>>> >> >>> > of
>>> >> >>> > the
>>> >> >>> > loop.
>>> >> >>> >
>>> >> >>> > Can you try making the filter part of the
>>> >> >>> > "VectorDataSet.add(old_sum,
>>> >> >>> > current)" operation?
>>> >> >>> >
>>> >> >>> > On Tue, Oct 14, 2014 at 5:05 PM, Maximilian Alber
>>> >> >>> > <al...@gmail.com> wrote:
>>> >> >>> >>
>>> >> >>> >> The deltaiteration calculates the cumulative sum (prefix sum)
>>> of
>>> >> >>> >> residual_2.
>>> >> >>> >>
>>> >> >>> >> I'm not sure if got you idea. I could add residual_2 to the
>>> >> >>> >> workset.
>>> >> >>> >> But
>>> >> >>> >> in that case I need to separate the "old_sum"(the sum up to
>>> now)
>>> >> >>> >> and
>>> >> >>> >> residual_2 each iteration. Actually I had that before, then I
>>> tried
>>> >> >>> >> the
>>> >> >>> >> Scala closure to clean up the code.
>>> >> >>> >>
>>> >> >>> >> On Tue, Oct 14, 2014 at 4:57 PM, Fabian Hueske <
>>> fhueske@apache.org>
>>> >> >>> >> wrote:
>>> >> >>> >>>
>>> >> >>> >>> Jep, I see you point.
>>> >> >>> >>> Conceptually, all data that changes and affects the result of
>>> an
>>> >> >>> >>> iteration should be part of the workset.
>>> >> >>> >>> Hence, the model kind of assumes that the datum
>>> "superStepNumber"
>>> >> >>> >>> should
>>> >> >>> >>> be part of the workset.
>>> >> >>> >>>
>>> >> >>> >>> I am not familiar with your application, but would it make
>>> sense
>>> >> >>> >>> to
>>> >> >>> >>> add
>>> >> >>> >>> the number as an additional attribute to the workset data set
>>> and
>>> >> >>> >>> increase
>>> >> >>> >>> it manually?
>>> >> >>> >>>
>>> >> >>> >>> 2014-10-14 16:45 GMT+02:00 Maximilian Alber
>>> >> >>> >>> <al...@gmail.com>:
>>> >> >>> >>>>
>>> >> >>> >>>> Ok, sounds true, but somehow I would like to execute it
>>> inside of
>>> >> >>> >>>> it. So
>>> >> >>> >>>> I probably need to do some nonsense work to make it part of
>>> it?
>>> >> >>> >>>>
>>> >> >>> >>>> On Tue, Oct 14, 2014 at 4:36 PM, Aljoscha Krettek
>>> >> >>> >>>> <al...@apache.org>
>>> >> >>> >>>> wrote:
>>> >> >>> >>>>>
>>> >> >>> >>>>> Dammit you beat me to it. But yes, this is exactly what I
>>> was
>>> >> >>> >>>>> just
>>> >> >>> >>>>> writing.
>>> >> >>> >>>>>
>>> >> >>> >>>>> On Tue, Oct 14, 2014 at 4:35 PM, Fabian Hueske
>>> >> >>> >>>>> <fh...@apache.org>
>>> >> >>> >>>>> wrote:
>>> >> >>> >>>>> > Hi,
>>> >> >>> >>>>> >
>>> >> >>> >>>>> > I'm not super familiar with the iterations, but my guess
>>> would
>>> >> >>> >>>>> > be
>>> >> >>> >>>>> > that the
>>> >> >>> >>>>> > filter is not evaluated as part of the iteration.
>>> >> >>> >>>>> > Since it is not connect to the workset, the filter is not
>>> part
>>> >> >>> >>>>> > of
>>> >> >>> >>>>> > the
>>> >> >>> >>>>> > loop
>>> >> >>> >>>>> > and evaluated once outside where no superset number is
>>> >> >>> >>>>> > available.
>>> >> >>> >>>>> > I guess, moving the filter outside of the loop gives the
>>> same
>>> >> >>> >>>>> > error.
>>> >> >>> >>>>> >
>>> >> >>> >>>>> > Cheers, Fabian
>>> >> >>> >>>>> >
>>> >> >>> >>>>> >
>>> >> >>> >>>>> >
>>> >> >>> >>>>> > 2014-10-14 16:18 GMT+02:00 Maximilian Alber
>>> >> >>> >>>>> > <al...@gmail.com>:
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >> Hmm or maybe not. With this code I get some strange
>>> error:
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >> def createPlan_find_center(env: ExecutionEnvironment) = {
>>> >> >>> >>>>> >> val X = env readTextFile config.xFile map
>>> >> >>> >>>>> >> {Vector.parseFromString(config.dimensions, _)};
>>> >> >>> >>>>> >> val residual = env readTextFile config.yFile map
>>> >> >>> >>>>> >> {Vector.parseFromString(_)};
>>> >> >>> >>>>> >> val randoms = env readTextFile config.randomFile map
>>> >> >>> >>>>> >> {Vector.parseFromString(_)}
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >> val residual_2 = residual * residual
>>> >> >>> >>>>> >> val ys = (residual_2 sumV) * (randoms filter {_.id == 0})
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >> val emptyDataSet = env.fromCollection[Vector](Seq())
>>> >> >>> >>>>> >> val sumVector =
>>> >> >>> >>>>> >> env.fromCollection(Seq(Vector.zeros(config.dimensions)))
>>> >> >>> >>>>> >> val cumSum = emptyDataSet.iterateDelta(sumVector,
>>> config.N,
>>> >> >>> >>>>> >> Array("id")) {
>>> >> >>> >>>>> >> (solutionset, old_sum) =>
>>> >> >>> >>>>> >> val current = residual_2 filter (new
>>> >> >>> >>>>> >> RichFilterFunction[Vector]{
>>> >> >>> >>>>> >> def filter(x: Vector) = x.id ==
>>> >> >>> >>>>> >> (getIterationRuntimeContext.getSuperstepNumber)
>>> >> >>> >>>>> >> })
>>> >> >>> >>>>> >> val sum = VectorDataSet.add(old_sum, current)
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >> (sum map (new RichMapFunction[Vector, Vector]{
>>> >> >>> >>>>> >> def map(x: Vector) = new
>>> >> >>> >>>>> >> Vector(getIterationRuntimeContext.getSuperstepNumber,
>>> >> >>> >>>>> >> x.values)
>>> >> >>> >>>>> >> }),
>>> >> >>> >>>>> >> sum)
>>> >> >>> >>>>> >> }
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >> Error:
>>> >> >>> >>>>> >> 10/14/2014 15:57:35: Job execution switched to status
>>> RUNNING
>>> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1)
>>> switched to
>>> >> >>> >>>>> >> SCHEDULED
>>> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1)
>>> switched to
>>> >> >>> >>>>> >> DEPLOYING
>>> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to
>>> >> >>> >>>>> >> SCHEDULED
>>> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to
>>> >> >>> >>>>> >> DEPLOYING
>>> >> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>>> >> >>> >>>>> >> (/tmp/tmpBhOsLd) -
>>> >> >>> >>>>> >> UTF-8) -> Map
>>> (org.apache.flink.api.scala.DataSet$$anon$1)
>>> >> >>> >>>>> >> (1/1)
>>> >> >>> >>>>> >> switched to
>>> >> >>> >>>>> >> SCHEDULED
>>> >> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>>> >> >>> >>>>> >> (/tmp/tmpBhOsLd) -
>>> >> >>> >>>>> >> UTF-8) -> Map
>>> (org.apache.flink.api.scala.DataSet$$anon$1)
>>> >> >>> >>>>> >> (1/1)
>>> >> >>> >>>>> >> switched to
>>> >> >>> >>>>> >> DEPLOYING
>>> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1)
>>> switched to
>>> >> >>> >>>>> >> RUNNING
>>> >> >>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration
>>> (Unnamed
>>> >> >>> >>>>> >> Delta
>>> >> >>> >>>>> >> Iteration)) (1/1) switched to SCHEDULED
>>> >> >>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration
>>> (Unnamed
>>> >> >>> >>>>> >> Delta
>>> >> >>> >>>>> >> Iteration)) (1/1) switched to DEPLOYING
>>> >> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>>> >> >>> >>>>> >> (/tmp/tmpBhOsLd) -
>>> >> >>> >>>>> >> UTF-8) -> Map
>>> (org.apache.flink.api.scala.DataSet$$anon$1)
>>> >> >>> >>>>> >> (1/1)
>>> >> >>> >>>>> >> switched to
>>> >> >>> >>>>> >> RUNNING
>>> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to
>>> >> >>> >>>>> >> RUNNING
>>> >> >>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration
>>> (Unnamed
>>> >> >>> >>>>> >> Delta
>>> >> >>> >>>>> >> Iteration)) (1/1) switched to RUNNING
>>> >> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>> >> >>> >>>>> >> ->
>>> >> >>> >>>>> >> Map
>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>> >> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched
>>> to
>>> >> >>> >>>>> >> SCHEDULED
>>> >> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>> >> >>> >>>>> >> ->
>>> >> >>> >>>>> >> Map
>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>> >> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched
>>> to
>>> >> >>> >>>>> >> DEPLOYING
>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>> >> >>> >>>>> >> ->
>>> >> >>> >>>>> >> Map
>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>> switched
>>> >> >>> >>>>> >> to
>>> >> >>> >>>>> >> SCHEDULED
>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>> >> >>> >>>>> >> ->
>>> >> >>> >>>>> >> Map
>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>> switched
>>> >> >>> >>>>> >> to
>>> >> >>> >>>>> >> DEPLOYING
>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>> >> >>> >>>>> >> ->
>>> >> >>> >>>>> >> Map
>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>> >> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched
>>> to
>>> >> >>> >>>>> >> RUNNING
>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>> >> >>> >>>>> >> ->
>>> >> >>> >>>>> >> Map
>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>> switched
>>> >> >>> >>>>> >> to
>>> >> >>> >>>>> >> RUNNING
>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>> >> >>> >>>>> >> ->
>>> >> >>> >>>>> >> Map
>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>> >> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched
>>> to
>>> >> >>> >>>>> >> FAILED
>>> >> >>> >>>>> >> java.lang.IllegalStateException: This stub is not part
>>> of an
>>> >> >>> >>>>> >> iteration
>>> >> >>> >>>>> >> step function.
>>> >> >>> >>>>> >> at
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
>>> >> >>> >>>>> >> at
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
>>> >> >>> >>>>> >> at
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
>>> >> >>> >>>>> >> at
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
>>> >> >>> >>>>> >> at
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
>>> >> >>> >>>>> >> at
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
>>> >> >>> >>>>> >> at
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
>>> >> >>> >>>>> >> at
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
>>> >> >>> >>>>> >> at
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
>>> >> >>> >>>>> >> at
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
>>> >> >>> >>>>> >> at
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
>>> >> >>> >>>>> >> at
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
>>> >> >>> >>>>> >> at java.lang.Thread.run(Thread.java:745)
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >> 10/14/2014 15:57:36: Job execution switched to status
>>> FAILING
>>> >> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to
>>> >> >>> >>>>> >> CANCELING
>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
>>> >> >>> >>>>> >> (/tmp/tmpBhOsLd) -
>>> >> >>> >>>>> >> UTF-8) -> Map
>>> (org.apache.flink.api.scala.DataSet$$anon$1)
>>> >> >>> >>>>> >> (1/1)
>>> >> >>> >>>>> >> switched to
>>> >> >>> >>>>> >> CANCELING
>>> >> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1)
>>> switched to
>>> >> >>> >>>>> >> CANCELING
>>> >> >>> >>>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration
>>> (Unnamed
>>> >> >>> >>>>> >> Delta
>>> >> >>> >>>>> >> Iteration)) (1/1) switched to CANCELING
>>> >> >>> >>>>> >> 10/14/2014 15:57:36: Map
>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1)
>>> >> >>> >>>>> >> (1/1) switched to CANCELED
>>> >> >>> >>>>> >> 10/14/2014 15:57:36: DataSink(TextOutputFormat
>>> >> >>> >>>>> >> (/tmp/tmplSYJ7S)
>>> >> >>> >>>>> >> -
>>> >> >>> >>>>> >> UTF-8)
>>> >> >>> >>>>> >> (1/1) switched to CANCELED
>>> >> >>> >>>>> >> 10/14/2014 15:57:36: Sync (WorksetIteration (Unnamed
>>> Delta
>>> >> >>> >>>>> >> Iteration))
>>> >> >>> >>>>> >> (1/1) switched to CANCELED
>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>> >> >>> >>>>> >> ->
>>> >> >>> >>>>> >> Map
>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>> switched
>>> >> >>> >>>>> >> to
>>> >> >>> >>>>> >> CANCELING
>>> >> >>> >>>>> >> 10/14/2014 15:57:36: Map
>>> >> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$2)
>>> >> >>> >>>>> >> (1/1)
>>> >> >>> >>>>> >> switched to CANCELED
>>> >> >>> >>>>> >> 10/14/2014 15:57:36: SolutionSet Delta (1/1) switched to
>>> >> >>> >>>>> >> CANCELED
>>> >> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to
>>> >> >>> >>>>> >> CANCELED
>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
>>> >> >>> >>>>> >> (/tmp/tmpBhOsLd) -
>>> >> >>> >>>>> >> UTF-8) -> Map
>>> (org.apache.flink.api.scala.DataSet$$anon$1)
>>> >> >>> >>>>> >> (1/1)
>>> >> >>> >>>>> >> switched to
>>> >> >>> >>>>> >> CANCELED
>>> >> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1)
>>> switched to
>>> >> >>> >>>>> >> CANCELED
>>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>> >> >>> >>>>> >> ->
>>> >> >>> >>>>> >> Map
>>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>> switched
>>> >> >>> >>>>> >> to
>>> >> >>> >>>>> >> CANCELED
>>> >> >>> >>>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration
>>> (Unnamed
>>> >> >>> >>>>> >> Delta
>>> >> >>> >>>>> >> Iteration)) (1/1) switched to CANCELED
>>> >> >>> >>>>> >> 10/14/2014 15:57:36: Job execution switched to status
>>> FAILED
>>> >> >>> >>>>> >> Error: The program execution failed:
>>> >> >>> >>>>> >> java.lang.IllegalStateException: This
>>> >> >>> >>>>> >> stub is not part of an iteration step function.
>>> >> >>> >>>>> >> at
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
>>> >> >>> >>>>> >> at
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
>>> >> >>> >>>>> >> at
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
>>> >> >>> >>>>> >> at
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
>>> >> >>> >>>>> >> at
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
>>> >> >>> >>>>> >> at
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
>>> >> >>> >>>>> >> at
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
>>> >> >>> >>>>> >> at
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
>>> >> >>> >>>>> >> at
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
>>> >> >>> >>>>> >> at
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
>>> >> >>> >>>>> >> at
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
>>> >> >>> >>>>> >> at
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >>
>>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
>>> >> >>> >>>>> >> at java.lang.Thread.run(Thread.java:745)
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >> On Tue, Oct 14, 2014 at 3:32 PM, Maximilian Alber
>>> >> >>> >>>>> >> <al...@gmail.com> wrote:
>>> >> >>> >>>>> >>>
>>> >> >>> >>>>> >>> Should work now.
>>> >> >>> >>>>> >>> Cheers
>>> >> >>> >>>>> >>>
>>> >> >>> >>>>> >>> On Fri, Oct 10, 2014 at 3:38 PM, Maximilian Alber
>>> >> >>> >>>>> >>> <al...@gmail.com> wrote:
>>> >> >>> >>>>> >>>>
>>> >> >>> >>>>> >>>> Ok, thanks.
>>> >> >>> >>>>> >>>> Please let me know when it is fixed.
>>> >> >>> >>>>> >>>>
>>> >> >>> >>>>> >>>> Cheers
>>> >> >>> >>>>> >>>> Max
>>> >> >>> >>>>> >>>>
>>> >> >>> >>>>> >>>> On Fri, Oct 10, 2014 at 1:34 PM, Stephan Ewen
>>> >> >>> >>>>> >>>> <se...@apache.org>
>>> >> >>> >>>>> >>>> wrote:
>>> >> >>> >>>>> >>>>>
>>> >> >>> >>>>> >>>>> Thank you, I will look into that...
>>> >> >>> >>>>> >>>>
>>> >> >>> >>>>> >>>>
>>> >> >>> >>>>> >>>
>>> >> >>> >>>>> >>
>>> >> >>> >>>>> >
>>> >> >>> >>>>
>>> >> >>> >>>>
>>> >> >>> >>>
>>> >> >>> >>
>>> >> >>> >
>>> >> >>
>>> >> >>
>>> >> >
>>> >
>>> >
>>>
>>
>>
>
Re: Forced to use Solution Set in Step Function
Posted by Maximilian Alber <al...@gmail.com>.
@Aljoscha Sorry, I just tried my workaround. There are some minor
conceptual bugs (caused by the id's...). I attached the new version.
Unfortunately there the compiler breaks. An issue is already open.
On Wed, Oct 15, 2014 at 11:53 AM, Maximilian Alber <
alber.maximilian@gmail.com> wrote:
> Ok.
>
> Here is a input variant:
> flink run -v bump_boost-0.1.jar -c bumpboost.Job x_file=X y_file=Y
> out_file=/tmp/tmpnWYamw random_file=random_file dimensions=1 N=100
> width_candidates_file=/tmp/tmpTJ4LDh iterations=30 multi_bump_boost=0
> gradient_descent_iterations=30 cache=False min_width=-4 max_width=6
> min_width_update=1e-08 max_width_update=10
>
> width_candidates_file is not needed by now. X, Y and random_file are
> attached.
>
> If you have problems, running it, let me know!
> Thanks!
>
> On Tue, Oct 14, 2014 at 10:40 PM, Aljoscha Krettek <al...@apache.org>
> wrote:
>
>> Transferring to variables: Unfortunately not possible right now but we
>> are working on it.
>>
>> On Tue, Oct 14, 2014 at 8:53 PM, Maximilian Alber
>> <al...@gmail.com> wrote:
>> > @ Stephan: Thanks! So I gonna switch!
>> >
>> > Sorry, my bad. I will provide you some sample by tomorrow morning.
>> >
>> > Yes. Workaround, because I cannot transfer them into variables, can I
>> by now
>> > (or will I ever)?
>> >
>> > Maybe some explanation to my solution:
>> > - X is for my a matrix of shape (N, d). Modeled in Flink as dataset of
>> > vectors. Each Vector has an ID which is the row number and an array with
>> > numbers, the actual row.
>> > - Y is for my a matrix of shape (N, 1) thus actually a column-vector.
>> > - old_sum is either a scalar if d == 1 or a row-vector aka matrix of
>> shape
>> > (1, N) or a Dataset with one Vector. (By now I have the convention to
>> give
>> > id -1 to them, comes from a former workaround...)
>> >
>> > The whole ID story comes from the fact that I need to know which stuff
>> > belongs together in mathematical operations (see my zip function). You
>> can
>> > look that up in util.scala, that's kind of my math library. I don't
>> want to
>> > imagine the mess in Java :)
>> >
>> > Cheers
>> > Max
>> >
>> >
>> >
>> > On Tue, Oct 14, 2014 at 6:28 PM, Aljoscha Krettek <al...@apache.org>
>> > wrote:
>> >>
>> >> Could you maybe also give some examples for the input expected by your
>> >> program?
>> >>
>> >> Also, the residual DataSet contains several Vectors while the sum (or
>> >> old_sum) DataSet is always only contains 1 Vector. Correct?
>> >>
>> >> On Tue, Oct 14, 2014 at 6:04 PM, Stephan Ewen <se...@apache.org>
>> wrote:
>> >> > BTW: The current master allows you to not join with the solution set,
>> >> > and
>> >> > only use it to accumulate data.
>> >> >
>> >> > On Tue, Oct 14, 2014 at 5:29 PM, Maximilian Alber
>> >> > <al...@gmail.com> wrote:
>> >> >>
>> >> >> Ok, that's possible too.
>> >> >>
>> >> >> VectorDataSet is just scala magic to ease my life (See below). If
>> you
>> >> >> want
>> >> >> to take a look, I appended the package. The main code is in
>> >> >> BumpBoost.scala.
>> >> >> In util.scala is the vector stuff.
>> >> >> Thanks!
>> >> >>
>> >> >> class VectorDataSet(X: DataSet[Vector]){
>> >> >> def + (Y: DataSet[Vector]) = VectorDataSet.add(X, Y)
>> >> >> def - (Y: DataSet[Vector]) = VectorDataSet.subt(X, Y)
>> >> >> def * (Y: DataSet[Vector]) = VectorDataSet.mult(X, Y)
>> >> >> def / (Y: DataSet[Vector]) = VectorDataSet.div(X, Y)
>> >> >>
>> >> >> def zip(Y: DataSet[Vector]) = VectorDataSet.zip(X, Y)
>> >> >> def dot(Y: DataSet[Vector]) = VectorDataSet.dot(X, Y)
>> >> >> def sumV() = VectorDataSet.sumV(X)
>> >> >> }
>> >> >> object VectorDataSet {
>> >> >> def add(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map
>> {x =>
>> >> >> x._1 + x._2}
>> >> >> def subt(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map
>> {x
>> >> >> =>
>> >> >> x._1 - x._2}
>> >> >> def mult(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map
>> {x
>> >> >> =>
>> >> >> x._1 * x._2}
>> >> >> def div(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map
>> {x =>
>> >> >> x._1 / x._2}
>> >> >>
>> >> >> def zip(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 join X2 where
>> >> >> "id"
>> >> >> equalTo "id"
>> >> >> def dot(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 * X2 sumV
>> >> >> def sumV(X1: DataSet[Vector]) = X1 reduce {_ + _}
>> >> >>
>> >> >> implicit def vectorDataSetWrapper(ds: DataSet[Vector]) = new
>> >> >> VectorDataSet(ds)
>> >> >> }
>> >> >>
>> >> >>
>> >> >> On Tue, Oct 14, 2014 at 5:21 PM, Aljoscha Krettek <
>> aljoscha@apache.org>
>> >> >> wrote:
>> >> >>>
>> >> >>> Maybe you could use the residual_2 data set as a broadcast dataset.
>> >> >>> i.e. make in available in the operation that adds the residual for
>> the
>> >> >>> current iteration number to the old_sum. (I'm not sure what the
>> >> >>> VectorDataSet.add() method does here). If you gave me the complete
>> >> >>> code I could try finding an elegant solution to that problem.
>> >> >>>
>> >> >>> On Tue, Oct 14, 2014 at 5:15 PM, Stephan Ewen <se...@apache.org>
>> >> >>> wrote:
>> >> >>> > That is an interesting case. Everything that is loop invariant is
>> >> >>> > computed
>> >> >>> > once outside the loop. You are looking for a way to make this
>> part
>> >> >>> > of
>> >> >>> > the
>> >> >>> > loop.
>> >> >>> >
>> >> >>> > Can you try making the filter part of the
>> >> >>> > "VectorDataSet.add(old_sum,
>> >> >>> > current)" operation?
>> >> >>> >
>> >> >>> > On Tue, Oct 14, 2014 at 5:05 PM, Maximilian Alber
>> >> >>> > <al...@gmail.com> wrote:
>> >> >>> >>
>> >> >>> >> The deltaiteration calculates the cumulative sum (prefix sum) of
>> >> >>> >> residual_2.
>> >> >>> >>
>> >> >>> >> I'm not sure if got you idea. I could add residual_2 to the
>> >> >>> >> workset.
>> >> >>> >> But
>> >> >>> >> in that case I need to separate the "old_sum"(the sum up to now)
>> >> >>> >> and
>> >> >>> >> residual_2 each iteration. Actually I had that before, then I
>> tried
>> >> >>> >> the
>> >> >>> >> Scala closure to clean up the code.
>> >> >>> >>
>> >> >>> >> On Tue, Oct 14, 2014 at 4:57 PM, Fabian Hueske <
>> fhueske@apache.org>
>> >> >>> >> wrote:
>> >> >>> >>>
>> >> >>> >>> Jep, I see you point.
>> >> >>> >>> Conceptually, all data that changes and affects the result of
>> an
>> >> >>> >>> iteration should be part of the workset.
>> >> >>> >>> Hence, the model kind of assumes that the datum
>> "superStepNumber"
>> >> >>> >>> should
>> >> >>> >>> be part of the workset.
>> >> >>> >>>
>> >> >>> >>> I am not familiar with your application, but would it make
>> sense
>> >> >>> >>> to
>> >> >>> >>> add
>> >> >>> >>> the number as an additional attribute to the workset data set
>> and
>> >> >>> >>> increase
>> >> >>> >>> it manually?
>> >> >>> >>>
>> >> >>> >>> 2014-10-14 16:45 GMT+02:00 Maximilian Alber
>> >> >>> >>> <al...@gmail.com>:
>> >> >>> >>>>
>> >> >>> >>>> Ok, sounds true, but somehow I would like to execute it
>> inside of
>> >> >>> >>>> it. So
>> >> >>> >>>> I probably need to do some nonsense work to make it part of
>> it?
>> >> >>> >>>>
>> >> >>> >>>> On Tue, Oct 14, 2014 at 4:36 PM, Aljoscha Krettek
>> >> >>> >>>> <al...@apache.org>
>> >> >>> >>>> wrote:
>> >> >>> >>>>>
>> >> >>> >>>>> Dammit you beat me to it. But yes, this is exactly what I was
>> >> >>> >>>>> just
>> >> >>> >>>>> writing.
>> >> >>> >>>>>
>> >> >>> >>>>> On Tue, Oct 14, 2014 at 4:35 PM, Fabian Hueske
>> >> >>> >>>>> <fh...@apache.org>
>> >> >>> >>>>> wrote:
>> >> >>> >>>>> > Hi,
>> >> >>> >>>>> >
>> >> >>> >>>>> > I'm not super familiar with the iterations, but my guess
>> would
>> >> >>> >>>>> > be
>> >> >>> >>>>> > that the
>> >> >>> >>>>> > filter is not evaluated as part of the iteration.
>> >> >>> >>>>> > Since it is not connect to the workset, the filter is not
>> part
>> >> >>> >>>>> > of
>> >> >>> >>>>> > the
>> >> >>> >>>>> > loop
>> >> >>> >>>>> > and evaluated once outside where no superset number is
>> >> >>> >>>>> > available.
>> >> >>> >>>>> > I guess, moving the filter outside of the loop gives the
>> same
>> >> >>> >>>>> > error.
>> >> >>> >>>>> >
>> >> >>> >>>>> > Cheers, Fabian
>> >> >>> >>>>> >
>> >> >>> >>>>> >
>> >> >>> >>>>> >
>> >> >>> >>>>> > 2014-10-14 16:18 GMT+02:00 Maximilian Alber
>> >> >>> >>>>> > <al...@gmail.com>:
>> >> >>> >>>>> >>
>> >> >>> >>>>> >> Hmm or maybe not. With this code I get some strange error:
>> >> >>> >>>>> >>
>> >> >>> >>>>> >> def createPlan_find_center(env: ExecutionEnvironment) = {
>> >> >>> >>>>> >> val X = env readTextFile config.xFile map
>> >> >>> >>>>> >> {Vector.parseFromString(config.dimensions, _)};
>> >> >>> >>>>> >> val residual = env readTextFile config.yFile map
>> >> >>> >>>>> >> {Vector.parseFromString(_)};
>> >> >>> >>>>> >> val randoms = env readTextFile config.randomFile map
>> >> >>> >>>>> >> {Vector.parseFromString(_)}
>> >> >>> >>>>> >>
>> >> >>> >>>>> >> val residual_2 = residual * residual
>> >> >>> >>>>> >> val ys = (residual_2 sumV) * (randoms filter {_.id == 0})
>> >> >>> >>>>> >>
>> >> >>> >>>>> >> val emptyDataSet = env.fromCollection[Vector](Seq())
>> >> >>> >>>>> >> val sumVector =
>> >> >>> >>>>> >> env.fromCollection(Seq(Vector.zeros(config.dimensions)))
>> >> >>> >>>>> >> val cumSum = emptyDataSet.iterateDelta(sumVector,
>> config.N,
>> >> >>> >>>>> >> Array("id")) {
>> >> >>> >>>>> >> (solutionset, old_sum) =>
>> >> >>> >>>>> >> val current = residual_2 filter (new
>> >> >>> >>>>> >> RichFilterFunction[Vector]{
>> >> >>> >>>>> >> def filter(x: Vector) = x.id ==
>> >> >>> >>>>> >> (getIterationRuntimeContext.getSuperstepNumber)
>> >> >>> >>>>> >> })
>> >> >>> >>>>> >> val sum = VectorDataSet.add(old_sum, current)
>> >> >>> >>>>> >>
>> >> >>> >>>>> >> (sum map (new RichMapFunction[Vector, Vector]{
>> >> >>> >>>>> >> def map(x: Vector) = new
>> >> >>> >>>>> >> Vector(getIterationRuntimeContext.getSuperstepNumber,
>> >> >>> >>>>> >> x.values)
>> >> >>> >>>>> >> }),
>> >> >>> >>>>> >> sum)
>> >> >>> >>>>> >> }
>> >> >>> >>>>> >>
>> >> >>> >>>>> >> Error:
>> >> >>> >>>>> >> 10/14/2014 15:57:35: Job execution switched to status
>> RUNNING
>> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched
>> to
>> >> >>> >>>>> >> SCHEDULED
>> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched
>> to
>> >> >>> >>>>> >> DEPLOYING
>> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to
>> >> >>> >>>>> >> SCHEDULED
>> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to
>> >> >>> >>>>> >> DEPLOYING
>> >> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>> >> >>> >>>>> >> (/tmp/tmpBhOsLd) -
>> >> >>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1)
>> >> >>> >>>>> >> (1/1)
>> >> >>> >>>>> >> switched to
>> >> >>> >>>>> >> SCHEDULED
>> >> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>> >> >>> >>>>> >> (/tmp/tmpBhOsLd) -
>> >> >>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1)
>> >> >>> >>>>> >> (1/1)
>> >> >>> >>>>> >> switched to
>> >> >>> >>>>> >> DEPLOYING
>> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched
>> to
>> >> >>> >>>>> >> RUNNING
>> >> >>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration
>> (Unnamed
>> >> >>> >>>>> >> Delta
>> >> >>> >>>>> >> Iteration)) (1/1) switched to SCHEDULED
>> >> >>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration
>> (Unnamed
>> >> >>> >>>>> >> Delta
>> >> >>> >>>>> >> Iteration)) (1/1) switched to DEPLOYING
>> >> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>> >> >>> >>>>> >> (/tmp/tmpBhOsLd) -
>> >> >>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1)
>> >> >>> >>>>> >> (1/1)
>> >> >>> >>>>> >> switched to
>> >> >>> >>>>> >> RUNNING
>> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to
>> >> >>> >>>>> >> RUNNING
>> >> >>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration
>> (Unnamed
>> >> >>> >>>>> >> Delta
>> >> >>> >>>>> >> Iteration)) (1/1) switched to RUNNING
>> >> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> >> >>> >>>>> >> ->
>> >> >>> >>>>> >> Map
>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>> >> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to
>> >> >>> >>>>> >> SCHEDULED
>> >> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> >> >>> >>>>> >> ->
>> >> >>> >>>>> >> Map
>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>> >> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to
>> >> >>> >>>>> >> DEPLOYING
>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> >> >>> >>>>> >> ->
>> >> >>> >>>>> >> Map
>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>> switched
>> >> >>> >>>>> >> to
>> >> >>> >>>>> >> SCHEDULED
>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> >> >>> >>>>> >> ->
>> >> >>> >>>>> >> Map
>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>> switched
>> >> >>> >>>>> >> to
>> >> >>> >>>>> >> DEPLOYING
>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> >> >>> >>>>> >> ->
>> >> >>> >>>>> >> Map
>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>> >> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to
>> >> >>> >>>>> >> RUNNING
>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> >> >>> >>>>> >> ->
>> >> >>> >>>>> >> Map
>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>> switched
>> >> >>> >>>>> >> to
>> >> >>> >>>>> >> RUNNING
>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> >> >>> >>>>> >> ->
>> >> >>> >>>>> >> Map
>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>> >> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to
>> >> >>> >>>>> >> FAILED
>> >> >>> >>>>> >> java.lang.IllegalStateException: This stub is not part of
>> an
>> >> >>> >>>>> >> iteration
>> >> >>> >>>>> >> step function.
>> >> >>> >>>>> >> at
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
>> >> >>> >>>>> >> at
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
>> >> >>> >>>>> >> at
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
>> >> >>> >>>>> >> at
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
>> >> >>> >>>>> >> at
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
>> >> >>> >>>>> >> at
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
>> >> >>> >>>>> >> at
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
>> >> >>> >>>>> >> at
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
>> >> >>> >>>>> >> at
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
>> >> >>> >>>>> >> at
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
>> >> >>> >>>>> >> at
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
>> >> >>> >>>>> >> at
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
>> >> >>> >>>>> >> at java.lang.Thread.run(Thread.java:745)
>> >> >>> >>>>> >>
>> >> >>> >>>>> >> 10/14/2014 15:57:36: Job execution switched to status
>> FAILING
>> >> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to
>> >> >>> >>>>> >> CANCELING
>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
>> >> >>> >>>>> >> (/tmp/tmpBhOsLd) -
>> >> >>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1)
>> >> >>> >>>>> >> (1/1)
>> >> >>> >>>>> >> switched to
>> >> >>> >>>>> >> CANCELING
>> >> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched
>> to
>> >> >>> >>>>> >> CANCELING
>> >> >>> >>>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration
>> (Unnamed
>> >> >>> >>>>> >> Delta
>> >> >>> >>>>> >> Iteration)) (1/1) switched to CANCELING
>> >> >>> >>>>> >> 10/14/2014 15:57:36: Map
>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1)
>> >> >>> >>>>> >> (1/1) switched to CANCELED
>> >> >>> >>>>> >> 10/14/2014 15:57:36: DataSink(TextOutputFormat
>> >> >>> >>>>> >> (/tmp/tmplSYJ7S)
>> >> >>> >>>>> >> -
>> >> >>> >>>>> >> UTF-8)
>> >> >>> >>>>> >> (1/1) switched to CANCELED
>> >> >>> >>>>> >> 10/14/2014 15:57:36: Sync (WorksetIteration (Unnamed Delta
>> >> >>> >>>>> >> Iteration))
>> >> >>> >>>>> >> (1/1) switched to CANCELED
>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> >> >>> >>>>> >> ->
>> >> >>> >>>>> >> Map
>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>> switched
>> >> >>> >>>>> >> to
>> >> >>> >>>>> >> CANCELING
>> >> >>> >>>>> >> 10/14/2014 15:57:36: Map
>> >> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$2)
>> >> >>> >>>>> >> (1/1)
>> >> >>> >>>>> >> switched to CANCELED
>> >> >>> >>>>> >> 10/14/2014 15:57:36: SolutionSet Delta (1/1) switched to
>> >> >>> >>>>> >> CANCELED
>> >> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to
>> >> >>> >>>>> >> CANCELED
>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
>> >> >>> >>>>> >> (/tmp/tmpBhOsLd) -
>> >> >>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1)
>> >> >>> >>>>> >> (1/1)
>> >> >>> >>>>> >> switched to
>> >> >>> >>>>> >> CANCELED
>> >> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched
>> to
>> >> >>> >>>>> >> CANCELED
>> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> >> >>> >>>>> >> ->
>> >> >>> >>>>> >> Map
>> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>> switched
>> >> >>> >>>>> >> to
>> >> >>> >>>>> >> CANCELED
>> >> >>> >>>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration
>> (Unnamed
>> >> >>> >>>>> >> Delta
>> >> >>> >>>>> >> Iteration)) (1/1) switched to CANCELED
>> >> >>> >>>>> >> 10/14/2014 15:57:36: Job execution switched to status
>> FAILED
>> >> >>> >>>>> >> Error: The program execution failed:
>> >> >>> >>>>> >> java.lang.IllegalStateException: This
>> >> >>> >>>>> >> stub is not part of an iteration step function.
>> >> >>> >>>>> >> at
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
>> >> >>> >>>>> >> at
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
>> >> >>> >>>>> >> at
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
>> >> >>> >>>>> >> at
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
>> >> >>> >>>>> >> at
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
>> >> >>> >>>>> >> at
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
>> >> >>> >>>>> >> at
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
>> >> >>> >>>>> >> at
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
>> >> >>> >>>>> >> at
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
>> >> >>> >>>>> >> at
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
>> >> >>> >>>>> >> at
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
>> >> >>> >>>>> >> at
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >>
>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
>> >> >>> >>>>> >> at java.lang.Thread.run(Thread.java:745)
>> >> >>> >>>>> >>
>> >> >>> >>>>> >> On Tue, Oct 14, 2014 at 3:32 PM, Maximilian Alber
>> >> >>> >>>>> >> <al...@gmail.com> wrote:
>> >> >>> >>>>> >>>
>> >> >>> >>>>> >>> Should work now.
>> >> >>> >>>>> >>> Cheers
>> >> >>> >>>>> >>>
>> >> >>> >>>>> >>> On Fri, Oct 10, 2014 at 3:38 PM, Maximilian Alber
>> >> >>> >>>>> >>> <al...@gmail.com> wrote:
>> >> >>> >>>>> >>>>
>> >> >>> >>>>> >>>> Ok, thanks.
>> >> >>> >>>>> >>>> Please let me know when it is fixed.
>> >> >>> >>>>> >>>>
>> >> >>> >>>>> >>>> Cheers
>> >> >>> >>>>> >>>> Max
>> >> >>> >>>>> >>>>
>> >> >>> >>>>> >>>> On Fri, Oct 10, 2014 at 1:34 PM, Stephan Ewen
>> >> >>> >>>>> >>>> <se...@apache.org>
>> >> >>> >>>>> >>>> wrote:
>> >> >>> >>>>> >>>>>
>> >> >>> >>>>> >>>>> Thank you, I will look into that...
>> >> >>> >>>>> >>>>
>> >> >>> >>>>> >>>>
>> >> >>> >>>>> >>>
>> >> >>> >>>>> >>
>> >> >>> >>>>> >
>> >> >>> >>>>
>> >> >>> >>>>
>> >> >>> >>>
>> >> >>> >>
>> >> >>> >
>> >> >>
>> >> >>
>> >> >
>> >
>> >
>>
>
>
Re: Forced to use Solution Set in Step Function
Posted by Maximilian Alber <al...@gmail.com>.
Ok.
Here is a input variant:
flink run -v bump_boost-0.1.jar -c bumpboost.Job x_file=X y_file=Y
out_file=/tmp/tmpnWYamw random_file=random_file dimensions=1 N=100
width_candidates_file=/tmp/tmpTJ4LDh iterations=30 multi_bump_boost=0
gradient_descent_iterations=30 cache=False min_width=-4 max_width=6
min_width_update=1e-08 max_width_update=10
width_candidates_file is not needed by now. X, Y and random_file are
attached.
If you have problems, running it, let me know!
Thanks!
On Tue, Oct 14, 2014 at 10:40 PM, Aljoscha Krettek <al...@apache.org>
wrote:
> Transferring to variables: Unfortunately not possible right now but we
> are working on it.
>
> On Tue, Oct 14, 2014 at 8:53 PM, Maximilian Alber
> <al...@gmail.com> wrote:
> > @ Stephan: Thanks! So I gonna switch!
> >
> > Sorry, my bad. I will provide you some sample by tomorrow morning.
> >
> > Yes. Workaround, because I cannot transfer them into variables, can I by
> now
> > (or will I ever)?
> >
> > Maybe some explanation to my solution:
> > - X is for my a matrix of shape (N, d). Modeled in Flink as dataset of
> > vectors. Each Vector has an ID which is the row number and an array with
> > numbers, the actual row.
> > - Y is for my a matrix of shape (N, 1) thus actually a column-vector.
> > - old_sum is either a scalar if d == 1 or a row-vector aka matrix of
> shape
> > (1, N) or a Dataset with one Vector. (By now I have the convention to
> give
> > id -1 to them, comes from a former workaround...)
> >
> > The whole ID story comes from the fact that I need to know which stuff
> > belongs together in mathematical operations (see my zip function). You
> can
> > look that up in util.scala, that's kind of my math library. I don't want
> to
> > imagine the mess in Java :)
> >
> > Cheers
> > Max
> >
> >
> >
> > On Tue, Oct 14, 2014 at 6:28 PM, Aljoscha Krettek <al...@apache.org>
> > wrote:
> >>
> >> Could you maybe also give some examples for the input expected by your
> >> program?
> >>
> >> Also, the residual DataSet contains several Vectors while the sum (or
> >> old_sum) DataSet is always only contains 1 Vector. Correct?
> >>
> >> On Tue, Oct 14, 2014 at 6:04 PM, Stephan Ewen <se...@apache.org> wrote:
> >> > BTW: The current master allows you to not join with the solution set,
> >> > and
> >> > only use it to accumulate data.
> >> >
> >> > On Tue, Oct 14, 2014 at 5:29 PM, Maximilian Alber
> >> > <al...@gmail.com> wrote:
> >> >>
> >> >> Ok, that's possible too.
> >> >>
> >> >> VectorDataSet is just scala magic to ease my life (See below). If you
> >> >> want
> >> >> to take a look, I appended the package. The main code is in
> >> >> BumpBoost.scala.
> >> >> In util.scala is the vector stuff.
> >> >> Thanks!
> >> >>
> >> >> class VectorDataSet(X: DataSet[Vector]){
> >> >> def + (Y: DataSet[Vector]) = VectorDataSet.add(X, Y)
> >> >> def - (Y: DataSet[Vector]) = VectorDataSet.subt(X, Y)
> >> >> def * (Y: DataSet[Vector]) = VectorDataSet.mult(X, Y)
> >> >> def / (Y: DataSet[Vector]) = VectorDataSet.div(X, Y)
> >> >>
> >> >> def zip(Y: DataSet[Vector]) = VectorDataSet.zip(X, Y)
> >> >> def dot(Y: DataSet[Vector]) = VectorDataSet.dot(X, Y)
> >> >> def sumV() = VectorDataSet.sumV(X)
> >> >> }
> >> >> object VectorDataSet {
> >> >> def add(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map {x
> =>
> >> >> x._1 + x._2}
> >> >> def subt(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map {x
> >> >> =>
> >> >> x._1 - x._2}
> >> >> def mult(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map {x
> >> >> =>
> >> >> x._1 * x._2}
> >> >> def div(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map {x
> =>
> >> >> x._1 / x._2}
> >> >>
> >> >> def zip(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 join X2 where
> >> >> "id"
> >> >> equalTo "id"
> >> >> def dot(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 * X2 sumV
> >> >> def sumV(X1: DataSet[Vector]) = X1 reduce {_ + _}
> >> >>
> >> >> implicit def vectorDataSetWrapper(ds: DataSet[Vector]) = new
> >> >> VectorDataSet(ds)
> >> >> }
> >> >>
> >> >>
> >> >> On Tue, Oct 14, 2014 at 5:21 PM, Aljoscha Krettek <
> aljoscha@apache.org>
> >> >> wrote:
> >> >>>
> >> >>> Maybe you could use the residual_2 data set as a broadcast dataset.
> >> >>> i.e. make in available in the operation that adds the residual for
> the
> >> >>> current iteration number to the old_sum. (I'm not sure what the
> >> >>> VectorDataSet.add() method does here). If you gave me the complete
> >> >>> code I could try finding an elegant solution to that problem.
> >> >>>
> >> >>> On Tue, Oct 14, 2014 at 5:15 PM, Stephan Ewen <se...@apache.org>
> >> >>> wrote:
> >> >>> > That is an interesting case. Everything that is loop invariant is
> >> >>> > computed
> >> >>> > once outside the loop. You are looking for a way to make this part
> >> >>> > of
> >> >>> > the
> >> >>> > loop.
> >> >>> >
> >> >>> > Can you try making the filter part of the
> >> >>> > "VectorDataSet.add(old_sum,
> >> >>> > current)" operation?
> >> >>> >
> >> >>> > On Tue, Oct 14, 2014 at 5:05 PM, Maximilian Alber
> >> >>> > <al...@gmail.com> wrote:
> >> >>> >>
> >> >>> >> The deltaiteration calculates the cumulative sum (prefix sum) of
> >> >>> >> residual_2.
> >> >>> >>
> >> >>> >> I'm not sure if got you idea. I could add residual_2 to the
> >> >>> >> workset.
> >> >>> >> But
> >> >>> >> in that case I need to separate the "old_sum"(the sum up to now)
> >> >>> >> and
> >> >>> >> residual_2 each iteration. Actually I had that before, then I
> tried
> >> >>> >> the
> >> >>> >> Scala closure to clean up the code.
> >> >>> >>
> >> >>> >> On Tue, Oct 14, 2014 at 4:57 PM, Fabian Hueske <
> fhueske@apache.org>
> >> >>> >> wrote:
> >> >>> >>>
> >> >>> >>> Jep, I see you point.
> >> >>> >>> Conceptually, all data that changes and affects the result of an
> >> >>> >>> iteration should be part of the workset.
> >> >>> >>> Hence, the model kind of assumes that the datum
> "superStepNumber"
> >> >>> >>> should
> >> >>> >>> be part of the workset.
> >> >>> >>>
> >> >>> >>> I am not familiar with your application, but would it make sense
> >> >>> >>> to
> >> >>> >>> add
> >> >>> >>> the number as an additional attribute to the workset data set
> and
> >> >>> >>> increase
> >> >>> >>> it manually?
> >> >>> >>>
> >> >>> >>> 2014-10-14 16:45 GMT+02:00 Maximilian Alber
> >> >>> >>> <al...@gmail.com>:
> >> >>> >>>>
> >> >>> >>>> Ok, sounds true, but somehow I would like to execute it inside
> of
> >> >>> >>>> it. So
> >> >>> >>>> I probably need to do some nonsense work to make it part of it?
> >> >>> >>>>
> >> >>> >>>> On Tue, Oct 14, 2014 at 4:36 PM, Aljoscha Krettek
> >> >>> >>>> <al...@apache.org>
> >> >>> >>>> wrote:
> >> >>> >>>>>
> >> >>> >>>>> Dammit you beat me to it. But yes, this is exactly what I was
> >> >>> >>>>> just
> >> >>> >>>>> writing.
> >> >>> >>>>>
> >> >>> >>>>> On Tue, Oct 14, 2014 at 4:35 PM, Fabian Hueske
> >> >>> >>>>> <fh...@apache.org>
> >> >>> >>>>> wrote:
> >> >>> >>>>> > Hi,
> >> >>> >>>>> >
> >> >>> >>>>> > I'm not super familiar with the iterations, but my guess
> would
> >> >>> >>>>> > be
> >> >>> >>>>> > that the
> >> >>> >>>>> > filter is not evaluated as part of the iteration.
> >> >>> >>>>> > Since it is not connect to the workset, the filter is not
> part
> >> >>> >>>>> > of
> >> >>> >>>>> > the
> >> >>> >>>>> > loop
> >> >>> >>>>> > and evaluated once outside where no superset number is
> >> >>> >>>>> > available.
> >> >>> >>>>> > I guess, moving the filter outside of the loop gives the
> same
> >> >>> >>>>> > error.
> >> >>> >>>>> >
> >> >>> >>>>> > Cheers, Fabian
> >> >>> >>>>> >
> >> >>> >>>>> >
> >> >>> >>>>> >
> >> >>> >>>>> > 2014-10-14 16:18 GMT+02:00 Maximilian Alber
> >> >>> >>>>> > <al...@gmail.com>:
> >> >>> >>>>> >>
> >> >>> >>>>> >> Hmm or maybe not. With this code I get some strange error:
> >> >>> >>>>> >>
> >> >>> >>>>> >> def createPlan_find_center(env: ExecutionEnvironment) = {
> >> >>> >>>>> >> val X = env readTextFile config.xFile map
> >> >>> >>>>> >> {Vector.parseFromString(config.dimensions, _)};
> >> >>> >>>>> >> val residual = env readTextFile config.yFile map
> >> >>> >>>>> >> {Vector.parseFromString(_)};
> >> >>> >>>>> >> val randoms = env readTextFile config.randomFile map
> >> >>> >>>>> >> {Vector.parseFromString(_)}
> >> >>> >>>>> >>
> >> >>> >>>>> >> val residual_2 = residual * residual
> >> >>> >>>>> >> val ys = (residual_2 sumV) * (randoms filter {_.id == 0})
> >> >>> >>>>> >>
> >> >>> >>>>> >> val emptyDataSet = env.fromCollection[Vector](Seq())
> >> >>> >>>>> >> val sumVector =
> >> >>> >>>>> >> env.fromCollection(Seq(Vector.zeros(config.dimensions)))
> >> >>> >>>>> >> val cumSum = emptyDataSet.iterateDelta(sumVector, config.N,
> >> >>> >>>>> >> Array("id")) {
> >> >>> >>>>> >> (solutionset, old_sum) =>
> >> >>> >>>>> >> val current = residual_2 filter (new
> >> >>> >>>>> >> RichFilterFunction[Vector]{
> >> >>> >>>>> >> def filter(x: Vector) = x.id ==
> >> >>> >>>>> >> (getIterationRuntimeContext.getSuperstepNumber)
> >> >>> >>>>> >> })
> >> >>> >>>>> >> val sum = VectorDataSet.add(old_sum, current)
> >> >>> >>>>> >>
> >> >>> >>>>> >> (sum map (new RichMapFunction[Vector, Vector]{
> >> >>> >>>>> >> def map(x: Vector) = new
> >> >>> >>>>> >> Vector(getIterationRuntimeContext.getSuperstepNumber,
> >> >>> >>>>> >> x.values)
> >> >>> >>>>> >> }),
> >> >>> >>>>> >> sum)
> >> >>> >>>>> >> }
> >> >>> >>>>> >>
> >> >>> >>>>> >> Error:
> >> >>> >>>>> >> 10/14/2014 15:57:35: Job execution switched to status
> RUNNING
> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched
> to
> >> >>> >>>>> >> SCHEDULED
> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched
> to
> >> >>> >>>>> >> DEPLOYING
> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to
> >> >>> >>>>> >> SCHEDULED
> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to
> >> >>> >>>>> >> DEPLOYING
> >> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
> >> >>> >>>>> >> (/tmp/tmpBhOsLd) -
> >> >>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1)
> >> >>> >>>>> >> (1/1)
> >> >>> >>>>> >> switched to
> >> >>> >>>>> >> SCHEDULED
> >> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
> >> >>> >>>>> >> (/tmp/tmpBhOsLd) -
> >> >>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1)
> >> >>> >>>>> >> (1/1)
> >> >>> >>>>> >> switched to
> >> >>> >>>>> >> DEPLOYING
> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched
> to
> >> >>> >>>>> >> RUNNING
> >> >>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration
> (Unnamed
> >> >>> >>>>> >> Delta
> >> >>> >>>>> >> Iteration)) (1/1) switched to SCHEDULED
> >> >>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration
> (Unnamed
> >> >>> >>>>> >> Delta
> >> >>> >>>>> >> Iteration)) (1/1) switched to DEPLOYING
> >> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
> >> >>> >>>>> >> (/tmp/tmpBhOsLd) -
> >> >>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1)
> >> >>> >>>>> >> (1/1)
> >> >>> >>>>> >> switched to
> >> >>> >>>>> >> RUNNING
> >> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to
> >> >>> >>>>> >> RUNNING
> >> >>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration
> (Unnamed
> >> >>> >>>>> >> Delta
> >> >>> >>>>> >> Iteration)) (1/1) switched to RUNNING
> >> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> >> >>> >>>>> >> ->
> >> >>> >>>>> >> Map
> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
> >> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to
> >> >>> >>>>> >> SCHEDULED
> >> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> >> >>> >>>>> >> ->
> >> >>> >>>>> >> Map
> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
> >> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to
> >> >>> >>>>> >> DEPLOYING
> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> >> >>> >>>>> >> ->
> >> >>> >>>>> >> Map
> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched
> >> >>> >>>>> >> to
> >> >>> >>>>> >> SCHEDULED
> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> >> >>> >>>>> >> ->
> >> >>> >>>>> >> Map
> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched
> >> >>> >>>>> >> to
> >> >>> >>>>> >> DEPLOYING
> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> >> >>> >>>>> >> ->
> >> >>> >>>>> >> Map
> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
> >> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to
> >> >>> >>>>> >> RUNNING
> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> >> >>> >>>>> >> ->
> >> >>> >>>>> >> Map
> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched
> >> >>> >>>>> >> to
> >> >>> >>>>> >> RUNNING
> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> >> >>> >>>>> >> ->
> >> >>> >>>>> >> Map
> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
> >> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to
> >> >>> >>>>> >> FAILED
> >> >>> >>>>> >> java.lang.IllegalStateException: This stub is not part of
> an
> >> >>> >>>>> >> iteration
> >> >>> >>>>> >> step function.
> >> >>> >>>>> >> at
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
> >> >>> >>>>> >> at
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
> >> >>> >>>>> >> at
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
> >> >>> >>>>> >> at
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
> >> >>> >>>>> >> at
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
> >> >>> >>>>> >> at
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
> >> >>> >>>>> >> at
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
> >> >>> >>>>> >> at
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
> >> >>> >>>>> >> at
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
> >> >>> >>>>> >> at
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
> >> >>> >>>>> >> at
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
> >> >>> >>>>> >> at
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
> >> >>> >>>>> >> at java.lang.Thread.run(Thread.java:745)
> >> >>> >>>>> >>
> >> >>> >>>>> >> 10/14/2014 15:57:36: Job execution switched to status
> FAILING
> >> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to
> >> >>> >>>>> >> CANCELING
> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
> >> >>> >>>>> >> (/tmp/tmpBhOsLd) -
> >> >>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1)
> >> >>> >>>>> >> (1/1)
> >> >>> >>>>> >> switched to
> >> >>> >>>>> >> CANCELING
> >> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched
> to
> >> >>> >>>>> >> CANCELING
> >> >>> >>>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration
> (Unnamed
> >> >>> >>>>> >> Delta
> >> >>> >>>>> >> Iteration)) (1/1) switched to CANCELING
> >> >>> >>>>> >> 10/14/2014 15:57:36: Map
> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1)
> >> >>> >>>>> >> (1/1) switched to CANCELED
> >> >>> >>>>> >> 10/14/2014 15:57:36: DataSink(TextOutputFormat
> >> >>> >>>>> >> (/tmp/tmplSYJ7S)
> >> >>> >>>>> >> -
> >> >>> >>>>> >> UTF-8)
> >> >>> >>>>> >> (1/1) switched to CANCELED
> >> >>> >>>>> >> 10/14/2014 15:57:36: Sync (WorksetIteration (Unnamed Delta
> >> >>> >>>>> >> Iteration))
> >> >>> >>>>> >> (1/1) switched to CANCELED
> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> >> >>> >>>>> >> ->
> >> >>> >>>>> >> Map
> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched
> >> >>> >>>>> >> to
> >> >>> >>>>> >> CANCELING
> >> >>> >>>>> >> 10/14/2014 15:57:36: Map
> >> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$2)
> >> >>> >>>>> >> (1/1)
> >> >>> >>>>> >> switched to CANCELED
> >> >>> >>>>> >> 10/14/2014 15:57:36: SolutionSet Delta (1/1) switched to
> >> >>> >>>>> >> CANCELED
> >> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to
> >> >>> >>>>> >> CANCELED
> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
> >> >>> >>>>> >> (/tmp/tmpBhOsLd) -
> >> >>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1)
> >> >>> >>>>> >> (1/1)
> >> >>> >>>>> >> switched to
> >> >>> >>>>> >> CANCELED
> >> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched
> to
> >> >>> >>>>> >> CANCELED
> >> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> >> >>> >>>>> >> ->
> >> >>> >>>>> >> Map
> >> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched
> >> >>> >>>>> >> to
> >> >>> >>>>> >> CANCELED
> >> >>> >>>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration
> (Unnamed
> >> >>> >>>>> >> Delta
> >> >>> >>>>> >> Iteration)) (1/1) switched to CANCELED
> >> >>> >>>>> >> 10/14/2014 15:57:36: Job execution switched to status
> FAILED
> >> >>> >>>>> >> Error: The program execution failed:
> >> >>> >>>>> >> java.lang.IllegalStateException: This
> >> >>> >>>>> >> stub is not part of an iteration step function.
> >> >>> >>>>> >> at
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
> >> >>> >>>>> >> at
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
> >> >>> >>>>> >> at
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
> >> >>> >>>>> >> at
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
> >> >>> >>>>> >> at
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
> >> >>> >>>>> >> at
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
> >> >>> >>>>> >> at
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
> >> >>> >>>>> >> at
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
> >> >>> >>>>> >> at
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
> >> >>> >>>>> >> at
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
> >> >>> >>>>> >> at
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
> >> >>> >>>>> >> at
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> >> >>> >>>>> >>
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
> >> >>> >>>>> >> at java.lang.Thread.run(Thread.java:745)
> >> >>> >>>>> >>
> >> >>> >>>>> >> On Tue, Oct 14, 2014 at 3:32 PM, Maximilian Alber
> >> >>> >>>>> >> <al...@gmail.com> wrote:
> >> >>> >>>>> >>>
> >> >>> >>>>> >>> Should work now.
> >> >>> >>>>> >>> Cheers
> >> >>> >>>>> >>>
> >> >>> >>>>> >>> On Fri, Oct 10, 2014 at 3:38 PM, Maximilian Alber
> >> >>> >>>>> >>> <al...@gmail.com> wrote:
> >> >>> >>>>> >>>>
> >> >>> >>>>> >>>> Ok, thanks.
> >> >>> >>>>> >>>> Please let me know when it is fixed.
> >> >>> >>>>> >>>>
> >> >>> >>>>> >>>> Cheers
> >> >>> >>>>> >>>> Max
> >> >>> >>>>> >>>>
> >> >>> >>>>> >>>> On Fri, Oct 10, 2014 at 1:34 PM, Stephan Ewen
> >> >>> >>>>> >>>> <se...@apache.org>
> >> >>> >>>>> >>>> wrote:
> >> >>> >>>>> >>>>>
> >> >>> >>>>> >>>>> Thank you, I will look into that...
> >> >>> >>>>> >>>>
> >> >>> >>>>> >>>>
> >> >>> >>>>> >>>
> >> >>> >>>>> >>
> >> >>> >>>>> >
> >> >>> >>>>
> >> >>> >>>>
> >> >>> >>>
> >> >>> >>
> >> >>> >
> >> >>
> >> >>
> >> >
> >
> >
>
Re: Forced to use Solution Set in Step Function
Posted by Aljoscha Krettek <al...@apache.org>.
Transferring to variables: Unfortunately not possible right now but we
are working on it.
On Tue, Oct 14, 2014 at 8:53 PM, Maximilian Alber
<al...@gmail.com> wrote:
> @ Stephan: Thanks! So I gonna switch!
>
> Sorry, my bad. I will provide you some sample by tomorrow morning.
>
> Yes. Workaround, because I cannot transfer them into variables, can I by now
> (or will I ever)?
>
> Maybe some explanation to my solution:
> - X is for my a matrix of shape (N, d). Modeled in Flink as dataset of
> vectors. Each Vector has an ID which is the row number and an array with
> numbers, the actual row.
> - Y is for my a matrix of shape (N, 1) thus actually a column-vector.
> - old_sum is either a scalar if d == 1 or a row-vector aka matrix of shape
> (1, N) or a Dataset with one Vector. (By now I have the convention to give
> id -1 to them, comes from a former workaround...)
>
> The whole ID story comes from the fact that I need to know which stuff
> belongs together in mathematical operations (see my zip function). You can
> look that up in util.scala, that's kind of my math library. I don't want to
> imagine the mess in Java :)
>
> Cheers
> Max
>
>
>
> On Tue, Oct 14, 2014 at 6:28 PM, Aljoscha Krettek <al...@apache.org>
> wrote:
>>
>> Could you maybe also give some examples for the input expected by your
>> program?
>>
>> Also, the residual DataSet contains several Vectors while the sum (or
>> old_sum) DataSet is always only contains 1 Vector. Correct?
>>
>> On Tue, Oct 14, 2014 at 6:04 PM, Stephan Ewen <se...@apache.org> wrote:
>> > BTW: The current master allows you to not join with the solution set,
>> > and
>> > only use it to accumulate data.
>> >
>> > On Tue, Oct 14, 2014 at 5:29 PM, Maximilian Alber
>> > <al...@gmail.com> wrote:
>> >>
>> >> Ok, that's possible too.
>> >>
>> >> VectorDataSet is just scala magic to ease my life (See below). If you
>> >> want
>> >> to take a look, I appended the package. The main code is in
>> >> BumpBoost.scala.
>> >> In util.scala is the vector stuff.
>> >> Thanks!
>> >>
>> >> class VectorDataSet(X: DataSet[Vector]){
>> >> def + (Y: DataSet[Vector]) = VectorDataSet.add(X, Y)
>> >> def - (Y: DataSet[Vector]) = VectorDataSet.subt(X, Y)
>> >> def * (Y: DataSet[Vector]) = VectorDataSet.mult(X, Y)
>> >> def / (Y: DataSet[Vector]) = VectorDataSet.div(X, Y)
>> >>
>> >> def zip(Y: DataSet[Vector]) = VectorDataSet.zip(X, Y)
>> >> def dot(Y: DataSet[Vector]) = VectorDataSet.dot(X, Y)
>> >> def sumV() = VectorDataSet.sumV(X)
>> >> }
>> >> object VectorDataSet {
>> >> def add(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map {x =>
>> >> x._1 + x._2}
>> >> def subt(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map {x
>> >> =>
>> >> x._1 - x._2}
>> >> def mult(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map {x
>> >> =>
>> >> x._1 * x._2}
>> >> def div(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map {x =>
>> >> x._1 / x._2}
>> >>
>> >> def zip(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 join X2 where
>> >> "id"
>> >> equalTo "id"
>> >> def dot(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 * X2 sumV
>> >> def sumV(X1: DataSet[Vector]) = X1 reduce {_ + _}
>> >>
>> >> implicit def vectorDataSetWrapper(ds: DataSet[Vector]) = new
>> >> VectorDataSet(ds)
>> >> }
>> >>
>> >>
>> >> On Tue, Oct 14, 2014 at 5:21 PM, Aljoscha Krettek <al...@apache.org>
>> >> wrote:
>> >>>
>> >>> Maybe you could use the residual_2 data set as a broadcast dataset.
>> >>> i.e. make in available in the operation that adds the residual for the
>> >>> current iteration number to the old_sum. (I'm not sure what the
>> >>> VectorDataSet.add() method does here). If you gave me the complete
>> >>> code I could try finding an elegant solution to that problem.
>> >>>
>> >>> On Tue, Oct 14, 2014 at 5:15 PM, Stephan Ewen <se...@apache.org>
>> >>> wrote:
>> >>> > That is an interesting case. Everything that is loop invariant is
>> >>> > computed
>> >>> > once outside the loop. You are looking for a way to make this part
>> >>> > of
>> >>> > the
>> >>> > loop.
>> >>> >
>> >>> > Can you try making the filter part of the
>> >>> > "VectorDataSet.add(old_sum,
>> >>> > current)" operation?
>> >>> >
>> >>> > On Tue, Oct 14, 2014 at 5:05 PM, Maximilian Alber
>> >>> > <al...@gmail.com> wrote:
>> >>> >>
>> >>> >> The deltaiteration calculates the cumulative sum (prefix sum) of
>> >>> >> residual_2.
>> >>> >>
>> >>> >> I'm not sure if got you idea. I could add residual_2 to the
>> >>> >> workset.
>> >>> >> But
>> >>> >> in that case I need to separate the "old_sum"(the sum up to now)
>> >>> >> and
>> >>> >> residual_2 each iteration. Actually I had that before, then I tried
>> >>> >> the
>> >>> >> Scala closure to clean up the code.
>> >>> >>
>> >>> >> On Tue, Oct 14, 2014 at 4:57 PM, Fabian Hueske <fh...@apache.org>
>> >>> >> wrote:
>> >>> >>>
>> >>> >>> Jep, I see you point.
>> >>> >>> Conceptually, all data that changes and affects the result of an
>> >>> >>> iteration should be part of the workset.
>> >>> >>> Hence, the model kind of assumes that the datum "superStepNumber"
>> >>> >>> should
>> >>> >>> be part of the workset.
>> >>> >>>
>> >>> >>> I am not familiar with your application, but would it make sense
>> >>> >>> to
>> >>> >>> add
>> >>> >>> the number as an additional attribute to the workset data set and
>> >>> >>> increase
>> >>> >>> it manually?
>> >>> >>>
>> >>> >>> 2014-10-14 16:45 GMT+02:00 Maximilian Alber
>> >>> >>> <al...@gmail.com>:
>> >>> >>>>
>> >>> >>>> Ok, sounds true, but somehow I would like to execute it inside of
>> >>> >>>> it. So
>> >>> >>>> I probably need to do some nonsense work to make it part of it?
>> >>> >>>>
>> >>> >>>> On Tue, Oct 14, 2014 at 4:36 PM, Aljoscha Krettek
>> >>> >>>> <al...@apache.org>
>> >>> >>>> wrote:
>> >>> >>>>>
>> >>> >>>>> Dammit you beat me to it. But yes, this is exactly what I was
>> >>> >>>>> just
>> >>> >>>>> writing.
>> >>> >>>>>
>> >>> >>>>> On Tue, Oct 14, 2014 at 4:35 PM, Fabian Hueske
>> >>> >>>>> <fh...@apache.org>
>> >>> >>>>> wrote:
>> >>> >>>>> > Hi,
>> >>> >>>>> >
>> >>> >>>>> > I'm not super familiar with the iterations, but my guess would
>> >>> >>>>> > be
>> >>> >>>>> > that the
>> >>> >>>>> > filter is not evaluated as part of the iteration.
>> >>> >>>>> > Since it is not connect to the workset, the filter is not part
>> >>> >>>>> > of
>> >>> >>>>> > the
>> >>> >>>>> > loop
>> >>> >>>>> > and evaluated once outside where no superset number is
>> >>> >>>>> > available.
>> >>> >>>>> > I guess, moving the filter outside of the loop gives the same
>> >>> >>>>> > error.
>> >>> >>>>> >
>> >>> >>>>> > Cheers, Fabian
>> >>> >>>>> >
>> >>> >>>>> >
>> >>> >>>>> >
>> >>> >>>>> > 2014-10-14 16:18 GMT+02:00 Maximilian Alber
>> >>> >>>>> > <al...@gmail.com>:
>> >>> >>>>> >>
>> >>> >>>>> >> Hmm or maybe not. With this code I get some strange error:
>> >>> >>>>> >>
>> >>> >>>>> >> def createPlan_find_center(env: ExecutionEnvironment) = {
>> >>> >>>>> >> val X = env readTextFile config.xFile map
>> >>> >>>>> >> {Vector.parseFromString(config.dimensions, _)};
>> >>> >>>>> >> val residual = env readTextFile config.yFile map
>> >>> >>>>> >> {Vector.parseFromString(_)};
>> >>> >>>>> >> val randoms = env readTextFile config.randomFile map
>> >>> >>>>> >> {Vector.parseFromString(_)}
>> >>> >>>>> >>
>> >>> >>>>> >> val residual_2 = residual * residual
>> >>> >>>>> >> val ys = (residual_2 sumV) * (randoms filter {_.id == 0})
>> >>> >>>>> >>
>> >>> >>>>> >> val emptyDataSet = env.fromCollection[Vector](Seq())
>> >>> >>>>> >> val sumVector =
>> >>> >>>>> >> env.fromCollection(Seq(Vector.zeros(config.dimensions)))
>> >>> >>>>> >> val cumSum = emptyDataSet.iterateDelta(sumVector, config.N,
>> >>> >>>>> >> Array("id")) {
>> >>> >>>>> >> (solutionset, old_sum) =>
>> >>> >>>>> >> val current = residual_2 filter (new
>> >>> >>>>> >> RichFilterFunction[Vector]{
>> >>> >>>>> >> def filter(x: Vector) = x.id ==
>> >>> >>>>> >> (getIterationRuntimeContext.getSuperstepNumber)
>> >>> >>>>> >> })
>> >>> >>>>> >> val sum = VectorDataSet.add(old_sum, current)
>> >>> >>>>> >>
>> >>> >>>>> >> (sum map (new RichMapFunction[Vector, Vector]{
>> >>> >>>>> >> def map(x: Vector) = new
>> >>> >>>>> >> Vector(getIterationRuntimeContext.getSuperstepNumber,
>> >>> >>>>> >> x.values)
>> >>> >>>>> >> }),
>> >>> >>>>> >> sum)
>> >>> >>>>> >> }
>> >>> >>>>> >>
>> >>> >>>>> >> Error:
>> >>> >>>>> >> 10/14/2014 15:57:35: Job execution switched to status RUNNING
>> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to
>> >>> >>>>> >> SCHEDULED
>> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to
>> >>> >>>>> >> DEPLOYING
>> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to
>> >>> >>>>> >> SCHEDULED
>> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to
>> >>> >>>>> >> DEPLOYING
>> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>> >>> >>>>> >> (/tmp/tmpBhOsLd) -
>> >>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1)
>> >>> >>>>> >> (1/1)
>> >>> >>>>> >> switched to
>> >>> >>>>> >> SCHEDULED
>> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>> >>> >>>>> >> (/tmp/tmpBhOsLd) -
>> >>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1)
>> >>> >>>>> >> (1/1)
>> >>> >>>>> >> switched to
>> >>> >>>>> >> DEPLOYING
>> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to
>> >>> >>>>> >> RUNNING
>> >>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed
>> >>> >>>>> >> Delta
>> >>> >>>>> >> Iteration)) (1/1) switched to SCHEDULED
>> >>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed
>> >>> >>>>> >> Delta
>> >>> >>>>> >> Iteration)) (1/1) switched to DEPLOYING
>> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>> >>> >>>>> >> (/tmp/tmpBhOsLd) -
>> >>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1)
>> >>> >>>>> >> (1/1)
>> >>> >>>>> >> switched to
>> >>> >>>>> >> RUNNING
>> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to
>> >>> >>>>> >> RUNNING
>> >>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed
>> >>> >>>>> >> Delta
>> >>> >>>>> >> Iteration)) (1/1) switched to RUNNING
>> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN
>> >>> >>>>> >>
>> >>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> >>> >>>>> >> ->
>> >>> >>>>> >> Map
>> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to
>> >>> >>>>> >> SCHEDULED
>> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN
>> >>> >>>>> >>
>> >>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> >>> >>>>> >> ->
>> >>> >>>>> >> Map
>> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to
>> >>> >>>>> >> DEPLOYING
>> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>> >>> >>>>> >>
>> >>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> >>> >>>>> >> ->
>> >>> >>>>> >> Map
>> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched
>> >>> >>>>> >> to
>> >>> >>>>> >> SCHEDULED
>> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>> >>> >>>>> >>
>> >>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> >>> >>>>> >> ->
>> >>> >>>>> >> Map
>> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched
>> >>> >>>>> >> to
>> >>> >>>>> >> DEPLOYING
>> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>> >>> >>>>> >>
>> >>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> >>> >>>>> >> ->
>> >>> >>>>> >> Map
>> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to
>> >>> >>>>> >> RUNNING
>> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>> >>> >>>>> >>
>> >>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> >>> >>>>> >> ->
>> >>> >>>>> >> Map
>> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched
>> >>> >>>>> >> to
>> >>> >>>>> >> RUNNING
>> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>> >>> >>>>> >>
>> >>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> >>> >>>>> >> ->
>> >>> >>>>> >> Map
>> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to
>> >>> >>>>> >> FAILED
>> >>> >>>>> >> java.lang.IllegalStateException: This stub is not part of an
>> >>> >>>>> >> iteration
>> >>> >>>>> >> step function.
>> >>> >>>>> >> at
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
>> >>> >>>>> >> at
>> >>> >>>>> >>
>> >>> >>>>> >> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
>> >>> >>>>> >> at
>> >>> >>>>> >>
>> >>> >>>>> >> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
>> >>> >>>>> >> at
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
>> >>> >>>>> >> at
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
>> >>> >>>>> >> at
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
>> >>> >>>>> >> at
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
>> >>> >>>>> >> at
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
>> >>> >>>>> >> at
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
>> >>> >>>>> >> at
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
>> >>> >>>>> >> at
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
>> >>> >>>>> >> at
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
>> >>> >>>>> >> at java.lang.Thread.run(Thread.java:745)
>> >>> >>>>> >>
>> >>> >>>>> >> 10/14/2014 15:57:36: Job execution switched to status FAILING
>> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to
>> >>> >>>>> >> CANCELING
>> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
>> >>> >>>>> >> (/tmp/tmpBhOsLd) -
>> >>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1)
>> >>> >>>>> >> (1/1)
>> >>> >>>>> >> switched to
>> >>> >>>>> >> CANCELING
>> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to
>> >>> >>>>> >> CANCELING
>> >>> >>>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed
>> >>> >>>>> >> Delta
>> >>> >>>>> >> Iteration)) (1/1) switched to CANCELING
>> >>> >>>>> >> 10/14/2014 15:57:36: Map
>> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1)
>> >>> >>>>> >> (1/1) switched to CANCELED
>> >>> >>>>> >> 10/14/2014 15:57:36: DataSink(TextOutputFormat
>> >>> >>>>> >> (/tmp/tmplSYJ7S)
>> >>> >>>>> >> -
>> >>> >>>>> >> UTF-8)
>> >>> >>>>> >> (1/1) switched to CANCELED
>> >>> >>>>> >> 10/14/2014 15:57:36: Sync (WorksetIteration (Unnamed Delta
>> >>> >>>>> >> Iteration))
>> >>> >>>>> >> (1/1) switched to CANCELED
>> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>> >>> >>>>> >>
>> >>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> >>> >>>>> >> ->
>> >>> >>>>> >> Map
>> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched
>> >>> >>>>> >> to
>> >>> >>>>> >> CANCELING
>> >>> >>>>> >> 10/14/2014 15:57:36: Map
>> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$2)
>> >>> >>>>> >> (1/1)
>> >>> >>>>> >> switched to CANCELED
>> >>> >>>>> >> 10/14/2014 15:57:36: SolutionSet Delta (1/1) switched to
>> >>> >>>>> >> CANCELED
>> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to
>> >>> >>>>> >> CANCELED
>> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
>> >>> >>>>> >> (/tmp/tmpBhOsLd) -
>> >>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1)
>> >>> >>>>> >> (1/1)
>> >>> >>>>> >> switched to
>> >>> >>>>> >> CANCELED
>> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to
>> >>> >>>>> >> CANCELED
>> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>> >>> >>>>> >>
>> >>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> >>> >>>>> >> ->
>> >>> >>>>> >> Map
>> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched
>> >>> >>>>> >> to
>> >>> >>>>> >> CANCELED
>> >>> >>>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed
>> >>> >>>>> >> Delta
>> >>> >>>>> >> Iteration)) (1/1) switched to CANCELED
>> >>> >>>>> >> 10/14/2014 15:57:36: Job execution switched to status FAILED
>> >>> >>>>> >> Error: The program execution failed:
>> >>> >>>>> >> java.lang.IllegalStateException: This
>> >>> >>>>> >> stub is not part of an iteration step function.
>> >>> >>>>> >> at
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
>> >>> >>>>> >> at
>> >>> >>>>> >>
>> >>> >>>>> >> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
>> >>> >>>>> >> at
>> >>> >>>>> >>
>> >>> >>>>> >> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
>> >>> >>>>> >> at
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
>> >>> >>>>> >> at
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
>> >>> >>>>> >> at
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
>> >>> >>>>> >> at
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
>> >>> >>>>> >> at
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
>> >>> >>>>> >> at
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
>> >>> >>>>> >> at
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
>> >>> >>>>> >> at
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
>> >>> >>>>> >> at
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >>
>> >>> >>>>> >> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
>> >>> >>>>> >> at java.lang.Thread.run(Thread.java:745)
>> >>> >>>>> >>
>> >>> >>>>> >> On Tue, Oct 14, 2014 at 3:32 PM, Maximilian Alber
>> >>> >>>>> >> <al...@gmail.com> wrote:
>> >>> >>>>> >>>
>> >>> >>>>> >>> Should work now.
>> >>> >>>>> >>> Cheers
>> >>> >>>>> >>>
>> >>> >>>>> >>> On Fri, Oct 10, 2014 at 3:38 PM, Maximilian Alber
>> >>> >>>>> >>> <al...@gmail.com> wrote:
>> >>> >>>>> >>>>
>> >>> >>>>> >>>> Ok, thanks.
>> >>> >>>>> >>>> Please let me know when it is fixed.
>> >>> >>>>> >>>>
>> >>> >>>>> >>>> Cheers
>> >>> >>>>> >>>> Max
>> >>> >>>>> >>>>
>> >>> >>>>> >>>> On Fri, Oct 10, 2014 at 1:34 PM, Stephan Ewen
>> >>> >>>>> >>>> <se...@apache.org>
>> >>> >>>>> >>>> wrote:
>> >>> >>>>> >>>>>
>> >>> >>>>> >>>>> Thank you, I will look into that...
>> >>> >>>>> >>>>
>> >>> >>>>> >>>>
>> >>> >>>>> >>>
>> >>> >>>>> >>
>> >>> >>>>> >
>> >>> >>>>
>> >>> >>>>
>> >>> >>>
>> >>> >>
>> >>> >
>> >>
>> >>
>> >
>
>
Re: Forced to use Solution Set in Step Function
Posted by Maximilian Alber <al...@gmail.com>.
@ Stephan: Thanks! So I gonna switch!
Sorry, my bad. I will provide you some sample by tomorrow morning.
Yes. Workaround, because I cannot transfer them into variables, can I by
now (or will I ever)?
Maybe some explanation to my solution:
- X is for my a matrix of shape (N, d). Modeled in Flink as dataset of
vectors. Each Vector has an ID which is the row number and an array with
numbers, the actual row.
- Y is for my a matrix of shape (N, 1) thus actually a column-vector.
- old_sum is either a scalar if d == 1 or a row-vector aka matrix of shape
(1, N) or a Dataset with one Vector. (By now I have the convention to give
id -1 to them, comes from a former workaround...)
The whole ID story comes from the fact that I need to know which stuff
belongs together in mathematical operations (see my zip function). You can
look that up in util.scala, that's kind of my math library. I don't want to
imagine the mess in Java :)
Cheers
Max
On Tue, Oct 14, 2014 at 6:28 PM, Aljoscha Krettek <al...@apache.org>
wrote:
> Could you maybe also give some examples for the input expected by your
> program?
>
> Also, the residual DataSet contains several Vectors while the sum (or
> old_sum) DataSet is always only contains 1 Vector. Correct?
>
> On Tue, Oct 14, 2014 at 6:04 PM, Stephan Ewen <se...@apache.org> wrote:
> > BTW: The current master allows you to not join with the solution set, and
> > only use it to accumulate data.
> >
> > On Tue, Oct 14, 2014 at 5:29 PM, Maximilian Alber
> > <al...@gmail.com> wrote:
> >>
> >> Ok, that's possible too.
> >>
> >> VectorDataSet is just scala magic to ease my life (See below). If you
> want
> >> to take a look, I appended the package. The main code is in
> BumpBoost.scala.
> >> In util.scala is the vector stuff.
> >> Thanks!
> >>
> >> class VectorDataSet(X: DataSet[Vector]){
> >> def + (Y: DataSet[Vector]) = VectorDataSet.add(X, Y)
> >> def - (Y: DataSet[Vector]) = VectorDataSet.subt(X, Y)
> >> def * (Y: DataSet[Vector]) = VectorDataSet.mult(X, Y)
> >> def / (Y: DataSet[Vector]) = VectorDataSet.div(X, Y)
> >>
> >> def zip(Y: DataSet[Vector]) = VectorDataSet.zip(X, Y)
> >> def dot(Y: DataSet[Vector]) = VectorDataSet.dot(X, Y)
> >> def sumV() = VectorDataSet.sumV(X)
> >> }
> >> object VectorDataSet {
> >> def add(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map {x =>
> >> x._1 + x._2}
> >> def subt(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map {x =>
> >> x._1 - x._2}
> >> def mult(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map {x =>
> >> x._1 * x._2}
> >> def div(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map {x =>
> >> x._1 / x._2}
> >>
> >> def zip(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 join X2 where
> "id"
> >> equalTo "id"
> >> def dot(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 * X2 sumV
> >> def sumV(X1: DataSet[Vector]) = X1 reduce {_ + _}
> >>
> >> implicit def vectorDataSetWrapper(ds: DataSet[Vector]) = new
> >> VectorDataSet(ds)
> >> }
> >>
> >>
> >> On Tue, Oct 14, 2014 at 5:21 PM, Aljoscha Krettek <al...@apache.org>
> >> wrote:
> >>>
> >>> Maybe you could use the residual_2 data set as a broadcast dataset.
> >>> i.e. make in available in the operation that adds the residual for the
> >>> current iteration number to the old_sum. (I'm not sure what the
> >>> VectorDataSet.add() method does here). If you gave me the complete
> >>> code I could try finding an elegant solution to that problem.
> >>>
> >>> On Tue, Oct 14, 2014 at 5:15 PM, Stephan Ewen <se...@apache.org>
> wrote:
> >>> > That is an interesting case. Everything that is loop invariant is
> >>> > computed
> >>> > once outside the loop. You are looking for a way to make this part of
> >>> > the
> >>> > loop.
> >>> >
> >>> > Can you try making the filter part of the "VectorDataSet.add(old_sum,
> >>> > current)" operation?
> >>> >
> >>> > On Tue, Oct 14, 2014 at 5:05 PM, Maximilian Alber
> >>> > <al...@gmail.com> wrote:
> >>> >>
> >>> >> The deltaiteration calculates the cumulative sum (prefix sum) of
> >>> >> residual_2.
> >>> >>
> >>> >> I'm not sure if got you idea. I could add residual_2 to the workset.
> >>> >> But
> >>> >> in that case I need to separate the "old_sum"(the sum up to now) and
> >>> >> residual_2 each iteration. Actually I had that before, then I tried
> >>> >> the
> >>> >> Scala closure to clean up the code.
> >>> >>
> >>> >> On Tue, Oct 14, 2014 at 4:57 PM, Fabian Hueske <fh...@apache.org>
> >>> >> wrote:
> >>> >>>
> >>> >>> Jep, I see you point.
> >>> >>> Conceptually, all data that changes and affects the result of an
> >>> >>> iteration should be part of the workset.
> >>> >>> Hence, the model kind of assumes that the datum "superStepNumber"
> >>> >>> should
> >>> >>> be part of the workset.
> >>> >>>
> >>> >>> I am not familiar with your application, but would it make sense to
> >>> >>> add
> >>> >>> the number as an additional attribute to the workset data set and
> >>> >>> increase
> >>> >>> it manually?
> >>> >>>
> >>> >>> 2014-10-14 16:45 GMT+02:00 Maximilian Alber
> >>> >>> <al...@gmail.com>:
> >>> >>>>
> >>> >>>> Ok, sounds true, but somehow I would like to execute it inside of
> >>> >>>> it. So
> >>> >>>> I probably need to do some nonsense work to make it part of it?
> >>> >>>>
> >>> >>>> On Tue, Oct 14, 2014 at 4:36 PM, Aljoscha Krettek
> >>> >>>> <al...@apache.org>
> >>> >>>> wrote:
> >>> >>>>>
> >>> >>>>> Dammit you beat me to it. But yes, this is exactly what I was
> just
> >>> >>>>> writing.
> >>> >>>>>
> >>> >>>>> On Tue, Oct 14, 2014 at 4:35 PM, Fabian Hueske <
> fhueske@apache.org>
> >>> >>>>> wrote:
> >>> >>>>> > Hi,
> >>> >>>>> >
> >>> >>>>> > I'm not super familiar with the iterations, but my guess would
> be
> >>> >>>>> > that the
> >>> >>>>> > filter is not evaluated as part of the iteration.
> >>> >>>>> > Since it is not connect to the workset, the filter is not part
> of
> >>> >>>>> > the
> >>> >>>>> > loop
> >>> >>>>> > and evaluated once outside where no superset number is
> available.
> >>> >>>>> > I guess, moving the filter outside of the loop gives the same
> >>> >>>>> > error.
> >>> >>>>> >
> >>> >>>>> > Cheers, Fabian
> >>> >>>>> >
> >>> >>>>> >
> >>> >>>>> >
> >>> >>>>> > 2014-10-14 16:18 GMT+02:00 Maximilian Alber
> >>> >>>>> > <al...@gmail.com>:
> >>> >>>>> >>
> >>> >>>>> >> Hmm or maybe not. With this code I get some strange error:
> >>> >>>>> >>
> >>> >>>>> >> def createPlan_find_center(env: ExecutionEnvironment) = {
> >>> >>>>> >> val X = env readTextFile config.xFile map
> >>> >>>>> >> {Vector.parseFromString(config.dimensions, _)};
> >>> >>>>> >> val residual = env readTextFile config.yFile map
> >>> >>>>> >> {Vector.parseFromString(_)};
> >>> >>>>> >> val randoms = env readTextFile config.randomFile map
> >>> >>>>> >> {Vector.parseFromString(_)}
> >>> >>>>> >>
> >>> >>>>> >> val residual_2 = residual * residual
> >>> >>>>> >> val ys = (residual_2 sumV) * (randoms filter {_.id == 0})
> >>> >>>>> >>
> >>> >>>>> >> val emptyDataSet = env.fromCollection[Vector](Seq())
> >>> >>>>> >> val sumVector =
> >>> >>>>> >> env.fromCollection(Seq(Vector.zeros(config.dimensions)))
> >>> >>>>> >> val cumSum = emptyDataSet.iterateDelta(sumVector, config.N,
> >>> >>>>> >> Array("id")) {
> >>> >>>>> >> (solutionset, old_sum) =>
> >>> >>>>> >> val current = residual_2 filter (new
> >>> >>>>> >> RichFilterFunction[Vector]{
> >>> >>>>> >> def filter(x: Vector) = x.id ==
> >>> >>>>> >> (getIterationRuntimeContext.getSuperstepNumber)
> >>> >>>>> >> })
> >>> >>>>> >> val sum = VectorDataSet.add(old_sum, current)
> >>> >>>>> >>
> >>> >>>>> >> (sum map (new RichMapFunction[Vector, Vector]{
> >>> >>>>> >> def map(x: Vector) = new
> >>> >>>>> >> Vector(getIterationRuntimeContext.getSuperstepNumber,
> x.values)
> >>> >>>>> >> }),
> >>> >>>>> >> sum)
> >>> >>>>> >> }
> >>> >>>>> >>
> >>> >>>>> >> Error:
> >>> >>>>> >> 10/14/2014 15:57:35: Job execution switched to status RUNNING
> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to
> >>> >>>>> >> SCHEDULED
> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to
> >>> >>>>> >> DEPLOYING
> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to
> SCHEDULED
> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to
> DEPLOYING
> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
> >>> >>>>> >> (/tmp/tmpBhOsLd) -
> >>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1)
> (1/1)
> >>> >>>>> >> switched to
> >>> >>>>> >> SCHEDULED
> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
> >>> >>>>> >> (/tmp/tmpBhOsLd) -
> >>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1)
> (1/1)
> >>> >>>>> >> switched to
> >>> >>>>> >> DEPLOYING
> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to
> >>> >>>>> >> RUNNING
> >>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed
> >>> >>>>> >> Delta
> >>> >>>>> >> Iteration)) (1/1) switched to SCHEDULED
> >>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed
> >>> >>>>> >> Delta
> >>> >>>>> >> Iteration)) (1/1) switched to DEPLOYING
> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
> >>> >>>>> >> (/tmp/tmpBhOsLd) -
> >>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1)
> (1/1)
> >>> >>>>> >> switched to
> >>> >>>>> >> RUNNING
> >>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to RUNNING
> >>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed
> >>> >>>>> >> Delta
> >>> >>>>> >> Iteration)) (1/1) switched to RUNNING
> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN
> >>> >>>>> >>
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> >>> >>>>> >> ->
> >>> >>>>> >> Map
> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to
> >>> >>>>> >> SCHEDULED
> >>> >>>>> >> 10/14/2014 15:57:35: CHAIN
> >>> >>>>> >>
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> >>> >>>>> >> ->
> >>> >>>>> >> Map
> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to
> >>> >>>>> >> DEPLOYING
> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
> >>> >>>>> >>
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> >>> >>>>> >> ->
> >>> >>>>> >> Map
> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
> >>> >>>>> >> SCHEDULED
> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
> >>> >>>>> >>
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> >>> >>>>> >> ->
> >>> >>>>> >> Map
> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
> >>> >>>>> >> DEPLOYING
> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
> >>> >>>>> >>
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> >>> >>>>> >> ->
> >>> >>>>> >> Map
> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to
> >>> >>>>> >> RUNNING
> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
> >>> >>>>> >>
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> >>> >>>>> >> ->
> >>> >>>>> >> Map
> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
> >>> >>>>> >> RUNNING
> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
> >>> >>>>> >>
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> >>> >>>>> >> ->
> >>> >>>>> >> Map
> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to
> >>> >>>>> >> FAILED
> >>> >>>>> >> java.lang.IllegalStateException: This stub is not part of an
> >>> >>>>> >> iteration
> >>> >>>>> >> step function.
> >>> >>>>> >> at
> >>> >>>>> >>
> >>> >>>>> >>
> >>> >>>>> >>
> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
> >>> >>>>> >> at
> >>> >>>>> >>
> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
> >>> >>>>> >> at
> >>> >>>>> >>
> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
> >>> >>>>> >> at
> >>> >>>>> >>
> >>> >>>>> >>
> >>> >>>>> >>
> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
> >>> >>>>> >> at
> >>> >>>>> >>
> >>> >>>>> >>
> >>> >>>>> >>
> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
> >>> >>>>> >> at
> >>> >>>>> >>
> >>> >>>>> >>
> >>> >>>>> >>
> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
> >>> >>>>> >> at
> >>> >>>>> >>
> >>> >>>>> >>
> >>> >>>>> >>
> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
> >>> >>>>> >> at
> >>> >>>>> >>
> >>> >>>>> >>
> >>> >>>>> >>
> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
> >>> >>>>> >> at
> >>> >>>>> >>
> >>> >>>>> >>
> >>> >>>>> >>
> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
> >>> >>>>> >> at
> >>> >>>>> >>
> >>> >>>>> >>
> >>> >>>>> >>
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
> >>> >>>>> >> at
> >>> >>>>> >>
> >>> >>>>> >>
> >>> >>>>> >>
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
> >>> >>>>> >> at
> >>> >>>>> >>
> >>> >>>>> >>
> >>> >>>>> >>
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
> >>> >>>>> >> at java.lang.Thread.run(Thread.java:745)
> >>> >>>>> >>
> >>> >>>>> >> 10/14/2014 15:57:36: Job execution switched to status FAILING
> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to
> CANCELING
> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
> >>> >>>>> >> (/tmp/tmpBhOsLd) -
> >>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1)
> (1/1)
> >>> >>>>> >> switched to
> >>> >>>>> >> CANCELING
> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to
> >>> >>>>> >> CANCELING
> >>> >>>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed
> >>> >>>>> >> Delta
> >>> >>>>> >> Iteration)) (1/1) switched to CANCELING
> >>> >>>>> >> 10/14/2014 15:57:36: Map
> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1)
> >>> >>>>> >> (1/1) switched to CANCELED
> >>> >>>>> >> 10/14/2014 15:57:36: DataSink(TextOutputFormat
> (/tmp/tmplSYJ7S)
> >>> >>>>> >> -
> >>> >>>>> >> UTF-8)
> >>> >>>>> >> (1/1) switched to CANCELED
> >>> >>>>> >> 10/14/2014 15:57:36: Sync (WorksetIteration (Unnamed Delta
> >>> >>>>> >> Iteration))
> >>> >>>>> >> (1/1) switched to CANCELED
> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
> >>> >>>>> >>
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> >>> >>>>> >> ->
> >>> >>>>> >> Map
> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
> >>> >>>>> >> CANCELING
> >>> >>>>> >> 10/14/2014 15:57:36: Map
> >>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$2)
> >>> >>>>> >> (1/1)
> >>> >>>>> >> switched to CANCELED
> >>> >>>>> >> 10/14/2014 15:57:36: SolutionSet Delta (1/1) switched to
> >>> >>>>> >> CANCELED
> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to
> CANCELED
> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
> >>> >>>>> >> (/tmp/tmpBhOsLd) -
> >>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1)
> (1/1)
> >>> >>>>> >> switched to
> >>> >>>>> >> CANCELED
> >>> >>>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to
> >>> >>>>> >> CANCELED
> >>> >>>>> >> 10/14/2014 15:57:36: CHAIN
> >>> >>>>> >>
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> >>> >>>>> >> ->
> >>> >>>>> >> Map
> >>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
> >>> >>>>> >> CANCELED
> >>> >>>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed
> >>> >>>>> >> Delta
> >>> >>>>> >> Iteration)) (1/1) switched to CANCELED
> >>> >>>>> >> 10/14/2014 15:57:36: Job execution switched to status FAILED
> >>> >>>>> >> Error: The program execution failed:
> >>> >>>>> >> java.lang.IllegalStateException: This
> >>> >>>>> >> stub is not part of an iteration step function.
> >>> >>>>> >> at
> >>> >>>>> >>
> >>> >>>>> >>
> >>> >>>>> >>
> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
> >>> >>>>> >> at
> >>> >>>>> >>
> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
> >>> >>>>> >> at
> >>> >>>>> >>
> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
> >>> >>>>> >> at
> >>> >>>>> >>
> >>> >>>>> >>
> >>> >>>>> >>
> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
> >>> >>>>> >> at
> >>> >>>>> >>
> >>> >>>>> >>
> >>> >>>>> >>
> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
> >>> >>>>> >> at
> >>> >>>>> >>
> >>> >>>>> >>
> >>> >>>>> >>
> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
> >>> >>>>> >> at
> >>> >>>>> >>
> >>> >>>>> >>
> >>> >>>>> >>
> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
> >>> >>>>> >> at
> >>> >>>>> >>
> >>> >>>>> >>
> >>> >>>>> >>
> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
> >>> >>>>> >> at
> >>> >>>>> >>
> >>> >>>>> >>
> >>> >>>>> >>
> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
> >>> >>>>> >> at
> >>> >>>>> >>
> >>> >>>>> >>
> >>> >>>>> >>
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
> >>> >>>>> >> at
> >>> >>>>> >>
> >>> >>>>> >>
> >>> >>>>> >>
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
> >>> >>>>> >> at
> >>> >>>>> >>
> >>> >>>>> >>
> >>> >>>>> >>
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
> >>> >>>>> >> at java.lang.Thread.run(Thread.java:745)
> >>> >>>>> >>
> >>> >>>>> >> On Tue, Oct 14, 2014 at 3:32 PM, Maximilian Alber
> >>> >>>>> >> <al...@gmail.com> wrote:
> >>> >>>>> >>>
> >>> >>>>> >>> Should work now.
> >>> >>>>> >>> Cheers
> >>> >>>>> >>>
> >>> >>>>> >>> On Fri, Oct 10, 2014 at 3:38 PM, Maximilian Alber
> >>> >>>>> >>> <al...@gmail.com> wrote:
> >>> >>>>> >>>>
> >>> >>>>> >>>> Ok, thanks.
> >>> >>>>> >>>> Please let me know when it is fixed.
> >>> >>>>> >>>>
> >>> >>>>> >>>> Cheers
> >>> >>>>> >>>> Max
> >>> >>>>> >>>>
> >>> >>>>> >>>> On Fri, Oct 10, 2014 at 1:34 PM, Stephan Ewen
> >>> >>>>> >>>> <se...@apache.org>
> >>> >>>>> >>>> wrote:
> >>> >>>>> >>>>>
> >>> >>>>> >>>>> Thank you, I will look into that...
> >>> >>>>> >>>>
> >>> >>>>> >>>>
> >>> >>>>> >>>
> >>> >>>>> >>
> >>> >>>>> >
> >>> >>>>
> >>> >>>>
> >>> >>>
> >>> >>
> >>> >
> >>
> >>
> >
>
Re: Forced to use Solution Set in Step Function
Posted by Aljoscha Krettek <al...@apache.org>.
Could you maybe also give some examples for the input expected by your program?
Also, the residual DataSet contains several Vectors while the sum (or
old_sum) DataSet is always only contains 1 Vector. Correct?
On Tue, Oct 14, 2014 at 6:04 PM, Stephan Ewen <se...@apache.org> wrote:
> BTW: The current master allows you to not join with the solution set, and
> only use it to accumulate data.
>
> On Tue, Oct 14, 2014 at 5:29 PM, Maximilian Alber
> <al...@gmail.com> wrote:
>>
>> Ok, that's possible too.
>>
>> VectorDataSet is just scala magic to ease my life (See below). If you want
>> to take a look, I appended the package. The main code is in BumpBoost.scala.
>> In util.scala is the vector stuff.
>> Thanks!
>>
>> class VectorDataSet(X: DataSet[Vector]){
>> def + (Y: DataSet[Vector]) = VectorDataSet.add(X, Y)
>> def - (Y: DataSet[Vector]) = VectorDataSet.subt(X, Y)
>> def * (Y: DataSet[Vector]) = VectorDataSet.mult(X, Y)
>> def / (Y: DataSet[Vector]) = VectorDataSet.div(X, Y)
>>
>> def zip(Y: DataSet[Vector]) = VectorDataSet.zip(X, Y)
>> def dot(Y: DataSet[Vector]) = VectorDataSet.dot(X, Y)
>> def sumV() = VectorDataSet.sumV(X)
>> }
>> object VectorDataSet {
>> def add(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map {x =>
>> x._1 + x._2}
>> def subt(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map {x =>
>> x._1 - x._2}
>> def mult(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map {x =>
>> x._1 * x._2}
>> def div(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map {x =>
>> x._1 / x._2}
>>
>> def zip(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 join X2 where "id"
>> equalTo "id"
>> def dot(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 * X2 sumV
>> def sumV(X1: DataSet[Vector]) = X1 reduce {_ + _}
>>
>> implicit def vectorDataSetWrapper(ds: DataSet[Vector]) = new
>> VectorDataSet(ds)
>> }
>>
>>
>> On Tue, Oct 14, 2014 at 5:21 PM, Aljoscha Krettek <al...@apache.org>
>> wrote:
>>>
>>> Maybe you could use the residual_2 data set as a broadcast dataset.
>>> i.e. make in available in the operation that adds the residual for the
>>> current iteration number to the old_sum. (I'm not sure what the
>>> VectorDataSet.add() method does here). If you gave me the complete
>>> code I could try finding an elegant solution to that problem.
>>>
>>> On Tue, Oct 14, 2014 at 5:15 PM, Stephan Ewen <se...@apache.org> wrote:
>>> > That is an interesting case. Everything that is loop invariant is
>>> > computed
>>> > once outside the loop. You are looking for a way to make this part of
>>> > the
>>> > loop.
>>> >
>>> > Can you try making the filter part of the "VectorDataSet.add(old_sum,
>>> > current)" operation?
>>> >
>>> > On Tue, Oct 14, 2014 at 5:05 PM, Maximilian Alber
>>> > <al...@gmail.com> wrote:
>>> >>
>>> >> The deltaiteration calculates the cumulative sum (prefix sum) of
>>> >> residual_2.
>>> >>
>>> >> I'm not sure if got you idea. I could add residual_2 to the workset.
>>> >> But
>>> >> in that case I need to separate the "old_sum"(the sum up to now) and
>>> >> residual_2 each iteration. Actually I had that before, then I tried
>>> >> the
>>> >> Scala closure to clean up the code.
>>> >>
>>> >> On Tue, Oct 14, 2014 at 4:57 PM, Fabian Hueske <fh...@apache.org>
>>> >> wrote:
>>> >>>
>>> >>> Jep, I see you point.
>>> >>> Conceptually, all data that changes and affects the result of an
>>> >>> iteration should be part of the workset.
>>> >>> Hence, the model kind of assumes that the datum "superStepNumber"
>>> >>> should
>>> >>> be part of the workset.
>>> >>>
>>> >>> I am not familiar with your application, but would it make sense to
>>> >>> add
>>> >>> the number as an additional attribute to the workset data set and
>>> >>> increase
>>> >>> it manually?
>>> >>>
>>> >>> 2014-10-14 16:45 GMT+02:00 Maximilian Alber
>>> >>> <al...@gmail.com>:
>>> >>>>
>>> >>>> Ok, sounds true, but somehow I would like to execute it inside of
>>> >>>> it. So
>>> >>>> I probably need to do some nonsense work to make it part of it?
>>> >>>>
>>> >>>> On Tue, Oct 14, 2014 at 4:36 PM, Aljoscha Krettek
>>> >>>> <al...@apache.org>
>>> >>>> wrote:
>>> >>>>>
>>> >>>>> Dammit you beat me to it. But yes, this is exactly what I was just
>>> >>>>> writing.
>>> >>>>>
>>> >>>>> On Tue, Oct 14, 2014 at 4:35 PM, Fabian Hueske <fh...@apache.org>
>>> >>>>> wrote:
>>> >>>>> > Hi,
>>> >>>>> >
>>> >>>>> > I'm not super familiar with the iterations, but my guess would be
>>> >>>>> > that the
>>> >>>>> > filter is not evaluated as part of the iteration.
>>> >>>>> > Since it is not connect to the workset, the filter is not part of
>>> >>>>> > the
>>> >>>>> > loop
>>> >>>>> > and evaluated once outside where no superset number is available.
>>> >>>>> > I guess, moving the filter outside of the loop gives the same
>>> >>>>> > error.
>>> >>>>> >
>>> >>>>> > Cheers, Fabian
>>> >>>>> >
>>> >>>>> >
>>> >>>>> >
>>> >>>>> > 2014-10-14 16:18 GMT+02:00 Maximilian Alber
>>> >>>>> > <al...@gmail.com>:
>>> >>>>> >>
>>> >>>>> >> Hmm or maybe not. With this code I get some strange error:
>>> >>>>> >>
>>> >>>>> >> def createPlan_find_center(env: ExecutionEnvironment) = {
>>> >>>>> >> val X = env readTextFile config.xFile map
>>> >>>>> >> {Vector.parseFromString(config.dimensions, _)};
>>> >>>>> >> val residual = env readTextFile config.yFile map
>>> >>>>> >> {Vector.parseFromString(_)};
>>> >>>>> >> val randoms = env readTextFile config.randomFile map
>>> >>>>> >> {Vector.parseFromString(_)}
>>> >>>>> >>
>>> >>>>> >> val residual_2 = residual * residual
>>> >>>>> >> val ys = (residual_2 sumV) * (randoms filter {_.id == 0})
>>> >>>>> >>
>>> >>>>> >> val emptyDataSet = env.fromCollection[Vector](Seq())
>>> >>>>> >> val sumVector =
>>> >>>>> >> env.fromCollection(Seq(Vector.zeros(config.dimensions)))
>>> >>>>> >> val cumSum = emptyDataSet.iterateDelta(sumVector, config.N,
>>> >>>>> >> Array("id")) {
>>> >>>>> >> (solutionset, old_sum) =>
>>> >>>>> >> val current = residual_2 filter (new
>>> >>>>> >> RichFilterFunction[Vector]{
>>> >>>>> >> def filter(x: Vector) = x.id ==
>>> >>>>> >> (getIterationRuntimeContext.getSuperstepNumber)
>>> >>>>> >> })
>>> >>>>> >> val sum = VectorDataSet.add(old_sum, current)
>>> >>>>> >>
>>> >>>>> >> (sum map (new RichMapFunction[Vector, Vector]{
>>> >>>>> >> def map(x: Vector) = new
>>> >>>>> >> Vector(getIterationRuntimeContext.getSuperstepNumber, x.values)
>>> >>>>> >> }),
>>> >>>>> >> sum)
>>> >>>>> >> }
>>> >>>>> >>
>>> >>>>> >> Error:
>>> >>>>> >> 10/14/2014 15:57:35: Job execution switched to status RUNNING
>>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to
>>> >>>>> >> SCHEDULED
>>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to
>>> >>>>> >> DEPLOYING
>>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to SCHEDULED
>>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to DEPLOYING
>>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>>> >>>>> >> (/tmp/tmpBhOsLd) -
>>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>> >>>>> >> switched to
>>> >>>>> >> SCHEDULED
>>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>>> >>>>> >> (/tmp/tmpBhOsLd) -
>>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>> >>>>> >> switched to
>>> >>>>> >> DEPLOYING
>>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to
>>> >>>>> >> RUNNING
>>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed
>>> >>>>> >> Delta
>>> >>>>> >> Iteration)) (1/1) switched to SCHEDULED
>>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed
>>> >>>>> >> Delta
>>> >>>>> >> Iteration)) (1/1) switched to DEPLOYING
>>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>>> >>>>> >> (/tmp/tmpBhOsLd) -
>>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>> >>>>> >> switched to
>>> >>>>> >> RUNNING
>>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to RUNNING
>>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed
>>> >>>>> >> Delta
>>> >>>>> >> Iteration)) (1/1) switched to RUNNING
>>> >>>>> >> 10/14/2014 15:57:35: CHAIN
>>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>> >>>>> >> ->
>>> >>>>> >> Map
>>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to
>>> >>>>> >> SCHEDULED
>>> >>>>> >> 10/14/2014 15:57:35: CHAIN
>>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>> >>>>> >> ->
>>> >>>>> >> Map
>>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to
>>> >>>>> >> DEPLOYING
>>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>> >>>>> >> ->
>>> >>>>> >> Map
>>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>>> >>>>> >> SCHEDULED
>>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>> >>>>> >> ->
>>> >>>>> >> Map
>>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>>> >>>>> >> DEPLOYING
>>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>> >>>>> >> ->
>>> >>>>> >> Map
>>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to
>>> >>>>> >> RUNNING
>>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>> >>>>> >> ->
>>> >>>>> >> Map
>>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>>> >>>>> >> RUNNING
>>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>> >>>>> >> ->
>>> >>>>> >> Map
>>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to
>>> >>>>> >> FAILED
>>> >>>>> >> java.lang.IllegalStateException: This stub is not part of an
>>> >>>>> >> iteration
>>> >>>>> >> step function.
>>> >>>>> >> at
>>> >>>>> >>
>>> >>>>> >>
>>> >>>>> >> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
>>> >>>>> >> at
>>> >>>>> >> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
>>> >>>>> >> at
>>> >>>>> >> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
>>> >>>>> >> at
>>> >>>>> >>
>>> >>>>> >>
>>> >>>>> >> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
>>> >>>>> >> at
>>> >>>>> >>
>>> >>>>> >>
>>> >>>>> >> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
>>> >>>>> >> at
>>> >>>>> >>
>>> >>>>> >>
>>> >>>>> >> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
>>> >>>>> >> at
>>> >>>>> >>
>>> >>>>> >>
>>> >>>>> >> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
>>> >>>>> >> at
>>> >>>>> >>
>>> >>>>> >>
>>> >>>>> >> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
>>> >>>>> >> at
>>> >>>>> >>
>>> >>>>> >>
>>> >>>>> >> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
>>> >>>>> >> at
>>> >>>>> >>
>>> >>>>> >>
>>> >>>>> >> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
>>> >>>>> >> at
>>> >>>>> >>
>>> >>>>> >>
>>> >>>>> >> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
>>> >>>>> >> at
>>> >>>>> >>
>>> >>>>> >>
>>> >>>>> >> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
>>> >>>>> >> at java.lang.Thread.run(Thread.java:745)
>>> >>>>> >>
>>> >>>>> >> 10/14/2014 15:57:36: Job execution switched to status FAILING
>>> >>>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to CANCELING
>>> >>>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
>>> >>>>> >> (/tmp/tmpBhOsLd) -
>>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>> >>>>> >> switched to
>>> >>>>> >> CANCELING
>>> >>>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to
>>> >>>>> >> CANCELING
>>> >>>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed
>>> >>>>> >> Delta
>>> >>>>> >> Iteration)) (1/1) switched to CANCELING
>>> >>>>> >> 10/14/2014 15:57:36: Map
>>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1)
>>> >>>>> >> (1/1) switched to CANCELED
>>> >>>>> >> 10/14/2014 15:57:36: DataSink(TextOutputFormat (/tmp/tmplSYJ7S)
>>> >>>>> >> -
>>> >>>>> >> UTF-8)
>>> >>>>> >> (1/1) switched to CANCELED
>>> >>>>> >> 10/14/2014 15:57:36: Sync (WorksetIteration (Unnamed Delta
>>> >>>>> >> Iteration))
>>> >>>>> >> (1/1) switched to CANCELED
>>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>> >>>>> >> ->
>>> >>>>> >> Map
>>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>>> >>>>> >> CANCELING
>>> >>>>> >> 10/14/2014 15:57:36: Map
>>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$2)
>>> >>>>> >> (1/1)
>>> >>>>> >> switched to CANCELED
>>> >>>>> >> 10/14/2014 15:57:36: SolutionSet Delta (1/1) switched to
>>> >>>>> >> CANCELED
>>> >>>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to CANCELED
>>> >>>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
>>> >>>>> >> (/tmp/tmpBhOsLd) -
>>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>> >>>>> >> switched to
>>> >>>>> >> CANCELED
>>> >>>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to
>>> >>>>> >> CANCELED
>>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>>> >>>>> >> ->
>>> >>>>> >> Map
>>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>>> >>>>> >> CANCELED
>>> >>>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed
>>> >>>>> >> Delta
>>> >>>>> >> Iteration)) (1/1) switched to CANCELED
>>> >>>>> >> 10/14/2014 15:57:36: Job execution switched to status FAILED
>>> >>>>> >> Error: The program execution failed:
>>> >>>>> >> java.lang.IllegalStateException: This
>>> >>>>> >> stub is not part of an iteration step function.
>>> >>>>> >> at
>>> >>>>> >>
>>> >>>>> >>
>>> >>>>> >> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
>>> >>>>> >> at
>>> >>>>> >> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
>>> >>>>> >> at
>>> >>>>> >> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
>>> >>>>> >> at
>>> >>>>> >>
>>> >>>>> >>
>>> >>>>> >> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
>>> >>>>> >> at
>>> >>>>> >>
>>> >>>>> >>
>>> >>>>> >> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
>>> >>>>> >> at
>>> >>>>> >>
>>> >>>>> >>
>>> >>>>> >> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
>>> >>>>> >> at
>>> >>>>> >>
>>> >>>>> >>
>>> >>>>> >> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
>>> >>>>> >> at
>>> >>>>> >>
>>> >>>>> >>
>>> >>>>> >> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
>>> >>>>> >> at
>>> >>>>> >>
>>> >>>>> >>
>>> >>>>> >> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
>>> >>>>> >> at
>>> >>>>> >>
>>> >>>>> >>
>>> >>>>> >> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
>>> >>>>> >> at
>>> >>>>> >>
>>> >>>>> >>
>>> >>>>> >> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
>>> >>>>> >> at
>>> >>>>> >>
>>> >>>>> >>
>>> >>>>> >> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
>>> >>>>> >> at java.lang.Thread.run(Thread.java:745)
>>> >>>>> >>
>>> >>>>> >> On Tue, Oct 14, 2014 at 3:32 PM, Maximilian Alber
>>> >>>>> >> <al...@gmail.com> wrote:
>>> >>>>> >>>
>>> >>>>> >>> Should work now.
>>> >>>>> >>> Cheers
>>> >>>>> >>>
>>> >>>>> >>> On Fri, Oct 10, 2014 at 3:38 PM, Maximilian Alber
>>> >>>>> >>> <al...@gmail.com> wrote:
>>> >>>>> >>>>
>>> >>>>> >>>> Ok, thanks.
>>> >>>>> >>>> Please let me know when it is fixed.
>>> >>>>> >>>>
>>> >>>>> >>>> Cheers
>>> >>>>> >>>> Max
>>> >>>>> >>>>
>>> >>>>> >>>> On Fri, Oct 10, 2014 at 1:34 PM, Stephan Ewen
>>> >>>>> >>>> <se...@apache.org>
>>> >>>>> >>>> wrote:
>>> >>>>> >>>>>
>>> >>>>> >>>>> Thank you, I will look into that...
>>> >>>>> >>>>
>>> >>>>> >>>>
>>> >>>>> >>>
>>> >>>>> >>
>>> >>>>> >
>>> >>>>
>>> >>>>
>>> >>>
>>> >>
>>> >
>>
>>
>
Re: Forced to use Solution Set in Step Function
Posted by Stephan Ewen <se...@apache.org>.
BTW: The current master allows you to not join with the solution set, and
only use it to accumulate data.
On Tue, Oct 14, 2014 at 5:29 PM, Maximilian Alber <
alber.maximilian@gmail.com> wrote:
> Ok, that's possible too.
>
> VectorDataSet is just scala magic to ease my life (See below). If you want
> to take a look, I appended the package. The main code is in
> BumpBoost.scala. In util.scala is the vector stuff.
> Thanks!
>
> class VectorDataSet(X: DataSet[Vector]){
> def + (Y: DataSet[Vector]) = VectorDataSet.add(X, Y)
> def - (Y: DataSet[Vector]) = VectorDataSet.subt(X, Y)
> def * (Y: DataSet[Vector]) = VectorDataSet.mult(X, Y)
> def / (Y: DataSet[Vector]) = VectorDataSet.div(X, Y)
>
> def zip(Y: DataSet[Vector]) = VectorDataSet.zip(X, Y)
> def dot(Y: DataSet[Vector]) = VectorDataSet.dot(X, Y)
> def sumV() = VectorDataSet.sumV(X)
> }
> object VectorDataSet {
> def add(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map {x =>
> x._1 + x._2}
> def subt(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map {x =>
> x._1 - x._2}
> def mult(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map {x =>
> x._1 * x._2}
> def div(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map {x =>
> x._1 / x._2}
>
> def zip(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 join X2 where "id"
> equalTo "id"
> def dot(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 * X2 sumV
> def sumV(X1: DataSet[Vector]) = X1 reduce {_ + _}
>
> implicit def vectorDataSetWrapper(ds: DataSet[Vector]) = new
> VectorDataSet(ds)
> }
>
>
> On Tue, Oct 14, 2014 at 5:21 PM, Aljoscha Krettek <al...@apache.org>
> wrote:
>
>> Maybe you could use the residual_2 data set as a broadcast dataset.
>> i.e. make in available in the operation that adds the residual for the
>> current iteration number to the old_sum. (I'm not sure what the
>> VectorDataSet.add() method does here). If you gave me the complete
>> code I could try finding an elegant solution to that problem.
>>
>> On Tue, Oct 14, 2014 at 5:15 PM, Stephan Ewen <se...@apache.org> wrote:
>> > That is an interesting case. Everything that is loop invariant is
>> computed
>> > once outside the loop. You are looking for a way to make this part of
>> the
>> > loop.
>> >
>> > Can you try making the filter part of the "VectorDataSet.add(old_sum,
>> > current)" operation?
>> >
>> > On Tue, Oct 14, 2014 at 5:05 PM, Maximilian Alber
>> > <al...@gmail.com> wrote:
>> >>
>> >> The deltaiteration calculates the cumulative sum (prefix sum) of
>> >> residual_2.
>> >>
>> >> I'm not sure if got you idea. I could add residual_2 to the workset.
>> But
>> >> in that case I need to separate the "old_sum"(the sum up to now) and
>> >> residual_2 each iteration. Actually I had that before, then I tried the
>> >> Scala closure to clean up the code.
>> >>
>> >> On Tue, Oct 14, 2014 at 4:57 PM, Fabian Hueske <fh...@apache.org>
>> wrote:
>> >>>
>> >>> Jep, I see you point.
>> >>> Conceptually, all data that changes and affects the result of an
>> >>> iteration should be part of the workset.
>> >>> Hence, the model kind of assumes that the datum "superStepNumber"
>> should
>> >>> be part of the workset.
>> >>>
>> >>> I am not familiar with your application, but would it make sense to
>> add
>> >>> the number as an additional attribute to the workset data set and
>> increase
>> >>> it manually?
>> >>>
>> >>> 2014-10-14 16:45 GMT+02:00 Maximilian Alber <
>> alber.maximilian@gmail.com>:
>> >>>>
>> >>>> Ok, sounds true, but somehow I would like to execute it inside of
>> it. So
>> >>>> I probably need to do some nonsense work to make it part of it?
>> >>>>
>> >>>> On Tue, Oct 14, 2014 at 4:36 PM, Aljoscha Krettek <
>> aljoscha@apache.org>
>> >>>> wrote:
>> >>>>>
>> >>>>> Dammit you beat me to it. But yes, this is exactly what I was just
>> >>>>> writing.
>> >>>>>
>> >>>>> On Tue, Oct 14, 2014 at 4:35 PM, Fabian Hueske <fh...@apache.org>
>> >>>>> wrote:
>> >>>>> > Hi,
>> >>>>> >
>> >>>>> > I'm not super familiar with the iterations, but my guess would be
>> >>>>> > that the
>> >>>>> > filter is not evaluated as part of the iteration.
>> >>>>> > Since it is not connect to the workset, the filter is not part of
>> the
>> >>>>> > loop
>> >>>>> > and evaluated once outside where no superset number is available.
>> >>>>> > I guess, moving the filter outside of the loop gives the same
>> error.
>> >>>>> >
>> >>>>> > Cheers, Fabian
>> >>>>> >
>> >>>>> >
>> >>>>> >
>> >>>>> > 2014-10-14 16:18 GMT+02:00 Maximilian Alber
>> >>>>> > <al...@gmail.com>:
>> >>>>> >>
>> >>>>> >> Hmm or maybe not. With this code I get some strange error:
>> >>>>> >>
>> >>>>> >> def createPlan_find_center(env: ExecutionEnvironment) = {
>> >>>>> >> val X = env readTextFile config.xFile map
>> >>>>> >> {Vector.parseFromString(config.dimensions, _)};
>> >>>>> >> val residual = env readTextFile config.yFile map
>> >>>>> >> {Vector.parseFromString(_)};
>> >>>>> >> val randoms = env readTextFile config.randomFile map
>> >>>>> >> {Vector.parseFromString(_)}
>> >>>>> >>
>> >>>>> >> val residual_2 = residual * residual
>> >>>>> >> val ys = (residual_2 sumV) * (randoms filter {_.id == 0})
>> >>>>> >>
>> >>>>> >> val emptyDataSet = env.fromCollection[Vector](Seq())
>> >>>>> >> val sumVector =
>> >>>>> >> env.fromCollection(Seq(Vector.zeros(config.dimensions)))
>> >>>>> >> val cumSum = emptyDataSet.iterateDelta(sumVector, config.N,
>> >>>>> >> Array("id")) {
>> >>>>> >> (solutionset, old_sum) =>
>> >>>>> >> val current = residual_2 filter (new
>> RichFilterFunction[Vector]{
>> >>>>> >> def filter(x: Vector) = x.id ==
>> >>>>> >> (getIterationRuntimeContext.getSuperstepNumber)
>> >>>>> >> })
>> >>>>> >> val sum = VectorDataSet.add(old_sum, current)
>> >>>>> >>
>> >>>>> >> (sum map (new RichMapFunction[Vector, Vector]{
>> >>>>> >> def map(x: Vector) = new
>> >>>>> >> Vector(getIterationRuntimeContext.getSuperstepNumber, x.values)
>> >>>>> >> }),
>> >>>>> >> sum)
>> >>>>> >> }
>> >>>>> >>
>> >>>>> >> Error:
>> >>>>> >> 10/14/2014 15:57:35: Job execution switched to status RUNNING
>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to
>> >>>>> >> SCHEDULED
>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to
>> >>>>> >> DEPLOYING
>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to SCHEDULED
>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to DEPLOYING
>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>> >>>>> >> (/tmp/tmpBhOsLd) -
>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>> >>>>> >> switched to
>> >>>>> >> SCHEDULED
>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>> >>>>> >> (/tmp/tmpBhOsLd) -
>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>> >>>>> >> switched to
>> >>>>> >> DEPLOYING
>> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to
>> RUNNING
>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed
>> Delta
>> >>>>> >> Iteration)) (1/1) switched to SCHEDULED
>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed
>> Delta
>> >>>>> >> Iteration)) (1/1) switched to DEPLOYING
>> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>> >>>>> >> (/tmp/tmpBhOsLd) -
>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>> >>>>> >> switched to
>> >>>>> >> RUNNING
>> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to RUNNING
>> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed
>> Delta
>> >>>>> >> Iteration)) (1/1) switched to RUNNING
>> >>>>> >> 10/14/2014 15:57:35: CHAIN
>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> ->
>> >>>>> >> Map
>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to
>> SCHEDULED
>> >>>>> >> 10/14/2014 15:57:35: CHAIN
>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> ->
>> >>>>> >> Map
>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to
>> DEPLOYING
>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> ->
>> >>>>> >> Map
>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>> >>>>> >> SCHEDULED
>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> ->
>> >>>>> >> Map
>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>> >>>>> >> DEPLOYING
>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> ->
>> >>>>> >> Map
>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to
>> RUNNING
>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> ->
>> >>>>> >> Map
>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>> >>>>> >> RUNNING
>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> ->
>> >>>>> >> Map
>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to FAILED
>> >>>>> >> java.lang.IllegalStateException: This stub is not part of an
>> >>>>> >> iteration
>> >>>>> >> step function.
>> >>>>> >> at
>> >>>>> >>
>> >>>>> >>
>> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
>> >>>>> >> at
>> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
>> >>>>> >> at
>> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
>> >>>>> >> at
>> >>>>> >>
>> >>>>> >>
>> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
>> >>>>> >> at
>> >>>>> >>
>> >>>>> >>
>> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
>> >>>>> >> at
>> >>>>> >>
>> >>>>> >>
>> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
>> >>>>> >> at
>> >>>>> >>
>> >>>>> >>
>> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
>> >>>>> >> at
>> >>>>> >>
>> >>>>> >>
>> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
>> >>>>> >> at
>> >>>>> >>
>> >>>>> >>
>> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
>> >>>>> >> at
>> >>>>> >>
>> >>>>> >>
>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
>> >>>>> >> at
>> >>>>> >>
>> >>>>> >>
>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
>> >>>>> >> at
>> >>>>> >>
>> >>>>> >>
>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
>> >>>>> >> at java.lang.Thread.run(Thread.java:745)
>> >>>>> >>
>> >>>>> >> 10/14/2014 15:57:36: Job execution switched to status FAILING
>> >>>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to CANCELING
>> >>>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
>> >>>>> >> (/tmp/tmpBhOsLd) -
>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>> >>>>> >> switched to
>> >>>>> >> CANCELING
>> >>>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to
>> >>>>> >> CANCELING
>> >>>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed
>> Delta
>> >>>>> >> Iteration)) (1/1) switched to CANCELING
>> >>>>> >> 10/14/2014 15:57:36: Map
>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1)
>> >>>>> >> (1/1) switched to CANCELED
>> >>>>> >> 10/14/2014 15:57:36: DataSink(TextOutputFormat (/tmp/tmplSYJ7S) -
>> >>>>> >> UTF-8)
>> >>>>> >> (1/1) switched to CANCELED
>> >>>>> >> 10/14/2014 15:57:36: Sync (WorksetIteration (Unnamed Delta
>> >>>>> >> Iteration))
>> >>>>> >> (1/1) switched to CANCELED
>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> ->
>> >>>>> >> Map
>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>> >>>>> >> CANCELING
>> >>>>> >> 10/14/2014 15:57:36: Map (bumpboost.BumpBoost$$anonfun$8$$anon$2)
>> >>>>> >> (1/1)
>> >>>>> >> switched to CANCELED
>> >>>>> >> 10/14/2014 15:57:36: SolutionSet Delta (1/1) switched to CANCELED
>> >>>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to CANCELED
>> >>>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
>> >>>>> >> (/tmp/tmpBhOsLd) -
>> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>> >>>>> >> switched to
>> >>>>> >> CANCELED
>> >>>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to
>> >>>>> >> CANCELED
>> >>>>> >> 10/14/2014 15:57:36: CHAIN
>> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
>> ->
>> >>>>> >> Map
>> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>> >>>>> >> CANCELED
>> >>>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed
>> Delta
>> >>>>> >> Iteration)) (1/1) switched to CANCELED
>> >>>>> >> 10/14/2014 15:57:36: Job execution switched to status FAILED
>> >>>>> >> Error: The program execution failed:
>> >>>>> >> java.lang.IllegalStateException: This
>> >>>>> >> stub is not part of an iteration step function.
>> >>>>> >> at
>> >>>>> >>
>> >>>>> >>
>> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
>> >>>>> >> at
>> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
>> >>>>> >> at
>> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
>> >>>>> >> at
>> >>>>> >>
>> >>>>> >>
>> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
>> >>>>> >> at
>> >>>>> >>
>> >>>>> >>
>> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
>> >>>>> >> at
>> >>>>> >>
>> >>>>> >>
>> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
>> >>>>> >> at
>> >>>>> >>
>> >>>>> >>
>> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
>> >>>>> >> at
>> >>>>> >>
>> >>>>> >>
>> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
>> >>>>> >> at
>> >>>>> >>
>> >>>>> >>
>> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
>> >>>>> >> at
>> >>>>> >>
>> >>>>> >>
>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
>> >>>>> >> at
>> >>>>> >>
>> >>>>> >>
>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
>> >>>>> >> at
>> >>>>> >>
>> >>>>> >>
>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
>> >>>>> >> at java.lang.Thread.run(Thread.java:745)
>> >>>>> >>
>> >>>>> >> On Tue, Oct 14, 2014 at 3:32 PM, Maximilian Alber
>> >>>>> >> <al...@gmail.com> wrote:
>> >>>>> >>>
>> >>>>> >>> Should work now.
>> >>>>> >>> Cheers
>> >>>>> >>>
>> >>>>> >>> On Fri, Oct 10, 2014 at 3:38 PM, Maximilian Alber
>> >>>>> >>> <al...@gmail.com> wrote:
>> >>>>> >>>>
>> >>>>> >>>> Ok, thanks.
>> >>>>> >>>> Please let me know when it is fixed.
>> >>>>> >>>>
>> >>>>> >>>> Cheers
>> >>>>> >>>> Max
>> >>>>> >>>>
>> >>>>> >>>> On Fri, Oct 10, 2014 at 1:34 PM, Stephan Ewen <
>> sewen@apache.org>
>> >>>>> >>>> wrote:
>> >>>>> >>>>>
>> >>>>> >>>>> Thank you, I will look into that...
>> >>>>> >>>>
>> >>>>> >>>>
>> >>>>> >>>
>> >>>>> >>
>> >>>>> >
>> >>>>
>> >>>>
>> >>>
>> >>
>> >
>>
>
>
Re: Forced to use Solution Set in Step Function
Posted by Maximilian Alber <al...@gmail.com>.
Ok, that's possible too.
VectorDataSet is just scala magic to ease my life (See below). If you want
to take a look, I appended the package. The main code is in
BumpBoost.scala. In util.scala is the vector stuff.
Thanks!
class VectorDataSet(X: DataSet[Vector]){
def + (Y: DataSet[Vector]) = VectorDataSet.add(X, Y)
def - (Y: DataSet[Vector]) = VectorDataSet.subt(X, Y)
def * (Y: DataSet[Vector]) = VectorDataSet.mult(X, Y)
def / (Y: DataSet[Vector]) = VectorDataSet.div(X, Y)
def zip(Y: DataSet[Vector]) = VectorDataSet.zip(X, Y)
def dot(Y: DataSet[Vector]) = VectorDataSet.dot(X, Y)
def sumV() = VectorDataSet.sumV(X)
}
object VectorDataSet {
def add(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map {x =>
x._1 + x._2}
def subt(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map {x =>
x._1 - x._2}
def mult(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map {x =>
x._1 * x._2}
def div(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 zip X2 map {x =>
x._1 / x._2}
def zip(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 join X2 where "id"
equalTo "id"
def dot(X1: DataSet[Vector], X2: DataSet[Vector]) = X1 * X2 sumV
def sumV(X1: DataSet[Vector]) = X1 reduce {_ + _}
implicit def vectorDataSetWrapper(ds: DataSet[Vector]) = new
VectorDataSet(ds)
}
On Tue, Oct 14, 2014 at 5:21 PM, Aljoscha Krettek <al...@apache.org>
wrote:
> Maybe you could use the residual_2 data set as a broadcast dataset.
> i.e. make in available in the operation that adds the residual for the
> current iteration number to the old_sum. (I'm not sure what the
> VectorDataSet.add() method does here). If you gave me the complete
> code I could try finding an elegant solution to that problem.
>
> On Tue, Oct 14, 2014 at 5:15 PM, Stephan Ewen <se...@apache.org> wrote:
> > That is an interesting case. Everything that is loop invariant is
> computed
> > once outside the loop. You are looking for a way to make this part of the
> > loop.
> >
> > Can you try making the filter part of the "VectorDataSet.add(old_sum,
> > current)" operation?
> >
> > On Tue, Oct 14, 2014 at 5:05 PM, Maximilian Alber
> > <al...@gmail.com> wrote:
> >>
> >> The deltaiteration calculates the cumulative sum (prefix sum) of
> >> residual_2.
> >>
> >> I'm not sure if got you idea. I could add residual_2 to the workset. But
> >> in that case I need to separate the "old_sum"(the sum up to now) and
> >> residual_2 each iteration. Actually I had that before, then I tried the
> >> Scala closure to clean up the code.
> >>
> >> On Tue, Oct 14, 2014 at 4:57 PM, Fabian Hueske <fh...@apache.org>
> wrote:
> >>>
> >>> Jep, I see you point.
> >>> Conceptually, all data that changes and affects the result of an
> >>> iteration should be part of the workset.
> >>> Hence, the model kind of assumes that the datum "superStepNumber"
> should
> >>> be part of the workset.
> >>>
> >>> I am not familiar with your application, but would it make sense to add
> >>> the number as an additional attribute to the workset data set and
> increase
> >>> it manually?
> >>>
> >>> 2014-10-14 16:45 GMT+02:00 Maximilian Alber <
> alber.maximilian@gmail.com>:
> >>>>
> >>>> Ok, sounds true, but somehow I would like to execute it inside of it.
> So
> >>>> I probably need to do some nonsense work to make it part of it?
> >>>>
> >>>> On Tue, Oct 14, 2014 at 4:36 PM, Aljoscha Krettek <
> aljoscha@apache.org>
> >>>> wrote:
> >>>>>
> >>>>> Dammit you beat me to it. But yes, this is exactly what I was just
> >>>>> writing.
> >>>>>
> >>>>> On Tue, Oct 14, 2014 at 4:35 PM, Fabian Hueske <fh...@apache.org>
> >>>>> wrote:
> >>>>> > Hi,
> >>>>> >
> >>>>> > I'm not super familiar with the iterations, but my guess would be
> >>>>> > that the
> >>>>> > filter is not evaluated as part of the iteration.
> >>>>> > Since it is not connect to the workset, the filter is not part of
> the
> >>>>> > loop
> >>>>> > and evaluated once outside where no superset number is available.
> >>>>> > I guess, moving the filter outside of the loop gives the same
> error.
> >>>>> >
> >>>>> > Cheers, Fabian
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > 2014-10-14 16:18 GMT+02:00 Maximilian Alber
> >>>>> > <al...@gmail.com>:
> >>>>> >>
> >>>>> >> Hmm or maybe not. With this code I get some strange error:
> >>>>> >>
> >>>>> >> def createPlan_find_center(env: ExecutionEnvironment) = {
> >>>>> >> val X = env readTextFile config.xFile map
> >>>>> >> {Vector.parseFromString(config.dimensions, _)};
> >>>>> >> val residual = env readTextFile config.yFile map
> >>>>> >> {Vector.parseFromString(_)};
> >>>>> >> val randoms = env readTextFile config.randomFile map
> >>>>> >> {Vector.parseFromString(_)}
> >>>>> >>
> >>>>> >> val residual_2 = residual * residual
> >>>>> >> val ys = (residual_2 sumV) * (randoms filter {_.id == 0})
> >>>>> >>
> >>>>> >> val emptyDataSet = env.fromCollection[Vector](Seq())
> >>>>> >> val sumVector =
> >>>>> >> env.fromCollection(Seq(Vector.zeros(config.dimensions)))
> >>>>> >> val cumSum = emptyDataSet.iterateDelta(sumVector, config.N,
> >>>>> >> Array("id")) {
> >>>>> >> (solutionset, old_sum) =>
> >>>>> >> val current = residual_2 filter (new
> RichFilterFunction[Vector]{
> >>>>> >> def filter(x: Vector) = x.id ==
> >>>>> >> (getIterationRuntimeContext.getSuperstepNumber)
> >>>>> >> })
> >>>>> >> val sum = VectorDataSet.add(old_sum, current)
> >>>>> >>
> >>>>> >> (sum map (new RichMapFunction[Vector, Vector]{
> >>>>> >> def map(x: Vector) = new
> >>>>> >> Vector(getIterationRuntimeContext.getSuperstepNumber, x.values)
> >>>>> >> }),
> >>>>> >> sum)
> >>>>> >> }
> >>>>> >>
> >>>>> >> Error:
> >>>>> >> 10/14/2014 15:57:35: Job execution switched to status RUNNING
> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to
> >>>>> >> SCHEDULED
> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to
> >>>>> >> DEPLOYING
> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to SCHEDULED
> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to DEPLOYING
> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
> >>>>> >> (/tmp/tmpBhOsLd) -
> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
> >>>>> >> switched to
> >>>>> >> SCHEDULED
> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
> >>>>> >> (/tmp/tmpBhOsLd) -
> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
> >>>>> >> switched to
> >>>>> >> DEPLOYING
> >>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to
> RUNNING
> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
> >>>>> >> Iteration)) (1/1) switched to SCHEDULED
> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
> >>>>> >> Iteration)) (1/1) switched to DEPLOYING
> >>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
> >>>>> >> (/tmp/tmpBhOsLd) -
> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
> >>>>> >> switched to
> >>>>> >> RUNNING
> >>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to RUNNING
> >>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
> >>>>> >> Iteration)) (1/1) switched to RUNNING
> >>>>> >> 10/14/2014 15:57:35: CHAIN
> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> ->
> >>>>> >> Map
> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to
> SCHEDULED
> >>>>> >> 10/14/2014 15:57:35: CHAIN
> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> ->
> >>>>> >> Map
> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to
> DEPLOYING
> >>>>> >> 10/14/2014 15:57:36: CHAIN
> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> ->
> >>>>> >> Map
> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
> >>>>> >> SCHEDULED
> >>>>> >> 10/14/2014 15:57:36: CHAIN
> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> ->
> >>>>> >> Map
> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
> >>>>> >> DEPLOYING
> >>>>> >> 10/14/2014 15:57:36: CHAIN
> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> ->
> >>>>> >> Map
> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to RUNNING
> >>>>> >> 10/14/2014 15:57:36: CHAIN
> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> ->
> >>>>> >> Map
> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
> >>>>> >> RUNNING
> >>>>> >> 10/14/2014 15:57:36: CHAIN
> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> ->
> >>>>> >> Map
> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
> >>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to FAILED
> >>>>> >> java.lang.IllegalStateException: This stub is not part of an
> >>>>> >> iteration
> >>>>> >> step function.
> >>>>> >> at
> >>>>> >>
> >>>>> >>
> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
> >>>>> >> at
> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
> >>>>> >> at
> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
> >>>>> >> at
> >>>>> >>
> >>>>> >>
> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
> >>>>> >> at
> >>>>> >>
> >>>>> >>
> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
> >>>>> >> at
> >>>>> >>
> >>>>> >>
> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
> >>>>> >> at
> >>>>> >>
> >>>>> >>
> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
> >>>>> >> at
> >>>>> >>
> >>>>> >>
> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
> >>>>> >> at
> >>>>> >>
> >>>>> >>
> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
> >>>>> >> at
> >>>>> >>
> >>>>> >>
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
> >>>>> >> at
> >>>>> >>
> >>>>> >>
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
> >>>>> >> at
> >>>>> >>
> >>>>> >>
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
> >>>>> >> at java.lang.Thread.run(Thread.java:745)
> >>>>> >>
> >>>>> >> 10/14/2014 15:57:36: Job execution switched to status FAILING
> >>>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to CANCELING
> >>>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
> >>>>> >> (/tmp/tmpBhOsLd) -
> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
> >>>>> >> switched to
> >>>>> >> CANCELING
> >>>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to
> >>>>> >> CANCELING
> >>>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed Delta
> >>>>> >> Iteration)) (1/1) switched to CANCELING
> >>>>> >> 10/14/2014 15:57:36: Map
> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1)
> >>>>> >> (1/1) switched to CANCELED
> >>>>> >> 10/14/2014 15:57:36: DataSink(TextOutputFormat (/tmp/tmplSYJ7S) -
> >>>>> >> UTF-8)
> >>>>> >> (1/1) switched to CANCELED
> >>>>> >> 10/14/2014 15:57:36: Sync (WorksetIteration (Unnamed Delta
> >>>>> >> Iteration))
> >>>>> >> (1/1) switched to CANCELED
> >>>>> >> 10/14/2014 15:57:36: CHAIN
> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> ->
> >>>>> >> Map
> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
> >>>>> >> CANCELING
> >>>>> >> 10/14/2014 15:57:36: Map (bumpboost.BumpBoost$$anonfun$8$$anon$2)
> >>>>> >> (1/1)
> >>>>> >> switched to CANCELED
> >>>>> >> 10/14/2014 15:57:36: SolutionSet Delta (1/1) switched to CANCELED
> >>>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to CANCELED
> >>>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
> >>>>> >> (/tmp/tmpBhOsLd) -
> >>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
> >>>>> >> switched to
> >>>>> >> CANCELED
> >>>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to
> >>>>> >> CANCELED
> >>>>> >> 10/14/2014 15:57:36: CHAIN
> >>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5)
> ->
> >>>>> >> Map
> >>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
> >>>>> >> CANCELED
> >>>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed Delta
> >>>>> >> Iteration)) (1/1) switched to CANCELED
> >>>>> >> 10/14/2014 15:57:36: Job execution switched to status FAILED
> >>>>> >> Error: The program execution failed:
> >>>>> >> java.lang.IllegalStateException: This
> >>>>> >> stub is not part of an iteration step function.
> >>>>> >> at
> >>>>> >>
> >>>>> >>
> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
> >>>>> >> at
> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
> >>>>> >> at
> bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
> >>>>> >> at
> >>>>> >>
> >>>>> >>
> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
> >>>>> >> at
> >>>>> >>
> >>>>> >>
> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
> >>>>> >> at
> >>>>> >>
> >>>>> >>
> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
> >>>>> >> at
> >>>>> >>
> >>>>> >>
> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
> >>>>> >> at
> >>>>> >>
> >>>>> >>
> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
> >>>>> >> at
> >>>>> >>
> >>>>> >>
> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
> >>>>> >> at
> >>>>> >>
> >>>>> >>
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
> >>>>> >> at
> >>>>> >>
> >>>>> >>
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
> >>>>> >> at
> >>>>> >>
> >>>>> >>
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
> >>>>> >> at java.lang.Thread.run(Thread.java:745)
> >>>>> >>
> >>>>> >> On Tue, Oct 14, 2014 at 3:32 PM, Maximilian Alber
> >>>>> >> <al...@gmail.com> wrote:
> >>>>> >>>
> >>>>> >>> Should work now.
> >>>>> >>> Cheers
> >>>>> >>>
> >>>>> >>> On Fri, Oct 10, 2014 at 3:38 PM, Maximilian Alber
> >>>>> >>> <al...@gmail.com> wrote:
> >>>>> >>>>
> >>>>> >>>> Ok, thanks.
> >>>>> >>>> Please let me know when it is fixed.
> >>>>> >>>>
> >>>>> >>>> Cheers
> >>>>> >>>> Max
> >>>>> >>>>
> >>>>> >>>> On Fri, Oct 10, 2014 at 1:34 PM, Stephan Ewen <sewen@apache.org
> >
> >>>>> >>>> wrote:
> >>>>> >>>>>
> >>>>> >>>>> Thank you, I will look into that...
> >>>>> >>>>
> >>>>> >>>>
> >>>>> >>>
> >>>>> >>
> >>>>> >
> >>>>
> >>>>
> >>>
> >>
> >
>
Re: Forced to use Solution Set in Step Function
Posted by Aljoscha Krettek <al...@apache.org>.
Maybe you could use the residual_2 data set as a broadcast dataset.
i.e. make in available in the operation that adds the residual for the
current iteration number to the old_sum. (I'm not sure what the
VectorDataSet.add() method does here). If you gave me the complete
code I could try finding an elegant solution to that problem.
On Tue, Oct 14, 2014 at 5:15 PM, Stephan Ewen <se...@apache.org> wrote:
> That is an interesting case. Everything that is loop invariant is computed
> once outside the loop. You are looking for a way to make this part of the
> loop.
>
> Can you try making the filter part of the "VectorDataSet.add(old_sum,
> current)" operation?
>
> On Tue, Oct 14, 2014 at 5:05 PM, Maximilian Alber
> <al...@gmail.com> wrote:
>>
>> The deltaiteration calculates the cumulative sum (prefix sum) of
>> residual_2.
>>
>> I'm not sure if got you idea. I could add residual_2 to the workset. But
>> in that case I need to separate the "old_sum"(the sum up to now) and
>> residual_2 each iteration. Actually I had that before, then I tried the
>> Scala closure to clean up the code.
>>
>> On Tue, Oct 14, 2014 at 4:57 PM, Fabian Hueske <fh...@apache.org> wrote:
>>>
>>> Jep, I see you point.
>>> Conceptually, all data that changes and affects the result of an
>>> iteration should be part of the workset.
>>> Hence, the model kind of assumes that the datum "superStepNumber" should
>>> be part of the workset.
>>>
>>> I am not familiar with your application, but would it make sense to add
>>> the number as an additional attribute to the workset data set and increase
>>> it manually?
>>>
>>> 2014-10-14 16:45 GMT+02:00 Maximilian Alber <al...@gmail.com>:
>>>>
>>>> Ok, sounds true, but somehow I would like to execute it inside of it. So
>>>> I probably need to do some nonsense work to make it part of it?
>>>>
>>>> On Tue, Oct 14, 2014 at 4:36 PM, Aljoscha Krettek <al...@apache.org>
>>>> wrote:
>>>>>
>>>>> Dammit you beat me to it. But yes, this is exactly what I was just
>>>>> writing.
>>>>>
>>>>> On Tue, Oct 14, 2014 at 4:35 PM, Fabian Hueske <fh...@apache.org>
>>>>> wrote:
>>>>> > Hi,
>>>>> >
>>>>> > I'm not super familiar with the iterations, but my guess would be
>>>>> > that the
>>>>> > filter is not evaluated as part of the iteration.
>>>>> > Since it is not connect to the workset, the filter is not part of the
>>>>> > loop
>>>>> > and evaluated once outside where no superset number is available.
>>>>> > I guess, moving the filter outside of the loop gives the same error.
>>>>> >
>>>>> > Cheers, Fabian
>>>>> >
>>>>> >
>>>>> >
>>>>> > 2014-10-14 16:18 GMT+02:00 Maximilian Alber
>>>>> > <al...@gmail.com>:
>>>>> >>
>>>>> >> Hmm or maybe not. With this code I get some strange error:
>>>>> >>
>>>>> >> def createPlan_find_center(env: ExecutionEnvironment) = {
>>>>> >> val X = env readTextFile config.xFile map
>>>>> >> {Vector.parseFromString(config.dimensions, _)};
>>>>> >> val residual = env readTextFile config.yFile map
>>>>> >> {Vector.parseFromString(_)};
>>>>> >> val randoms = env readTextFile config.randomFile map
>>>>> >> {Vector.parseFromString(_)}
>>>>> >>
>>>>> >> val residual_2 = residual * residual
>>>>> >> val ys = (residual_2 sumV) * (randoms filter {_.id == 0})
>>>>> >>
>>>>> >> val emptyDataSet = env.fromCollection[Vector](Seq())
>>>>> >> val sumVector =
>>>>> >> env.fromCollection(Seq(Vector.zeros(config.dimensions)))
>>>>> >> val cumSum = emptyDataSet.iterateDelta(sumVector, config.N,
>>>>> >> Array("id")) {
>>>>> >> (solutionset, old_sum) =>
>>>>> >> val current = residual_2 filter (new RichFilterFunction[Vector]{
>>>>> >> def filter(x: Vector) = x.id ==
>>>>> >> (getIterationRuntimeContext.getSuperstepNumber)
>>>>> >> })
>>>>> >> val sum = VectorDataSet.add(old_sum, current)
>>>>> >>
>>>>> >> (sum map (new RichMapFunction[Vector, Vector]{
>>>>> >> def map(x: Vector) = new
>>>>> >> Vector(getIterationRuntimeContext.getSuperstepNumber, x.values)
>>>>> >> }),
>>>>> >> sum)
>>>>> >> }
>>>>> >>
>>>>> >> Error:
>>>>> >> 10/14/2014 15:57:35: Job execution switched to status RUNNING
>>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to
>>>>> >> SCHEDULED
>>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to
>>>>> >> DEPLOYING
>>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to SCHEDULED
>>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to DEPLOYING
>>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>>>>> >> (/tmp/tmpBhOsLd) -
>>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>>>> >> switched to
>>>>> >> SCHEDULED
>>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>>>>> >> (/tmp/tmpBhOsLd) -
>>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>>>> >> switched to
>>>>> >> DEPLOYING
>>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to RUNNING
>>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
>>>>> >> Iteration)) (1/1) switched to SCHEDULED
>>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
>>>>> >> Iteration)) (1/1) switched to DEPLOYING
>>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>>>>> >> (/tmp/tmpBhOsLd) -
>>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>>>> >> switched to
>>>>> >> RUNNING
>>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to RUNNING
>>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
>>>>> >> Iteration)) (1/1) switched to RUNNING
>>>>> >> 10/14/2014 15:57:35: CHAIN
>>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>>> >> Map
>>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to SCHEDULED
>>>>> >> 10/14/2014 15:57:35: CHAIN
>>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>>> >> Map
>>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to DEPLOYING
>>>>> >> 10/14/2014 15:57:36: CHAIN
>>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>>> >> Map
>>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>>>>> >> SCHEDULED
>>>>> >> 10/14/2014 15:57:36: CHAIN
>>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>>> >> Map
>>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>>>>> >> DEPLOYING
>>>>> >> 10/14/2014 15:57:36: CHAIN
>>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>>> >> Map
>>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to RUNNING
>>>>> >> 10/14/2014 15:57:36: CHAIN
>>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>>> >> Map
>>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>>>>> >> RUNNING
>>>>> >> 10/14/2014 15:57:36: CHAIN
>>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>>> >> Map
>>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to FAILED
>>>>> >> java.lang.IllegalStateException: This stub is not part of an
>>>>> >> iteration
>>>>> >> step function.
>>>>> >> at
>>>>> >>
>>>>> >> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
>>>>> >> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
>>>>> >> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
>>>>> >> at
>>>>> >>
>>>>> >> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
>>>>> >> at
>>>>> >>
>>>>> >> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
>>>>> >> at
>>>>> >>
>>>>> >> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
>>>>> >> at
>>>>> >>
>>>>> >> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
>>>>> >> at
>>>>> >>
>>>>> >> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
>>>>> >> at
>>>>> >>
>>>>> >> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
>>>>> >> at
>>>>> >>
>>>>> >> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
>>>>> >> at
>>>>> >>
>>>>> >> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
>>>>> >> at
>>>>> >>
>>>>> >> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
>>>>> >> at java.lang.Thread.run(Thread.java:745)
>>>>> >>
>>>>> >> 10/14/2014 15:57:36: Job execution switched to status FAILING
>>>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to CANCELING
>>>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
>>>>> >> (/tmp/tmpBhOsLd) -
>>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>>>> >> switched to
>>>>> >> CANCELING
>>>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to
>>>>> >> CANCELING
>>>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed Delta
>>>>> >> Iteration)) (1/1) switched to CANCELING
>>>>> >> 10/14/2014 15:57:36: Map
>>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1)
>>>>> >> (1/1) switched to CANCELED
>>>>> >> 10/14/2014 15:57:36: DataSink(TextOutputFormat (/tmp/tmplSYJ7S) -
>>>>> >> UTF-8)
>>>>> >> (1/1) switched to CANCELED
>>>>> >> 10/14/2014 15:57:36: Sync (WorksetIteration (Unnamed Delta
>>>>> >> Iteration))
>>>>> >> (1/1) switched to CANCELED
>>>>> >> 10/14/2014 15:57:36: CHAIN
>>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>>> >> Map
>>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>>>>> >> CANCELING
>>>>> >> 10/14/2014 15:57:36: Map (bumpboost.BumpBoost$$anonfun$8$$anon$2)
>>>>> >> (1/1)
>>>>> >> switched to CANCELED
>>>>> >> 10/14/2014 15:57:36: SolutionSet Delta (1/1) switched to CANCELED
>>>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to CANCELED
>>>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
>>>>> >> (/tmp/tmpBhOsLd) -
>>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>>>> >> switched to
>>>>> >> CANCELED
>>>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to
>>>>> >> CANCELED
>>>>> >> 10/14/2014 15:57:36: CHAIN
>>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>>> >> Map
>>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>>>>> >> CANCELED
>>>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed Delta
>>>>> >> Iteration)) (1/1) switched to CANCELED
>>>>> >> 10/14/2014 15:57:36: Job execution switched to status FAILED
>>>>> >> Error: The program execution failed:
>>>>> >> java.lang.IllegalStateException: This
>>>>> >> stub is not part of an iteration step function.
>>>>> >> at
>>>>> >>
>>>>> >> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
>>>>> >> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
>>>>> >> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
>>>>> >> at
>>>>> >>
>>>>> >> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
>>>>> >> at
>>>>> >>
>>>>> >> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
>>>>> >> at
>>>>> >>
>>>>> >> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
>>>>> >> at
>>>>> >>
>>>>> >> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
>>>>> >> at
>>>>> >>
>>>>> >> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
>>>>> >> at
>>>>> >>
>>>>> >> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
>>>>> >> at
>>>>> >>
>>>>> >> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
>>>>> >> at
>>>>> >>
>>>>> >> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
>>>>> >> at
>>>>> >>
>>>>> >> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
>>>>> >> at java.lang.Thread.run(Thread.java:745)
>>>>> >>
>>>>> >> On Tue, Oct 14, 2014 at 3:32 PM, Maximilian Alber
>>>>> >> <al...@gmail.com> wrote:
>>>>> >>>
>>>>> >>> Should work now.
>>>>> >>> Cheers
>>>>> >>>
>>>>> >>> On Fri, Oct 10, 2014 at 3:38 PM, Maximilian Alber
>>>>> >>> <al...@gmail.com> wrote:
>>>>> >>>>
>>>>> >>>> Ok, thanks.
>>>>> >>>> Please let me know when it is fixed.
>>>>> >>>>
>>>>> >>>> Cheers
>>>>> >>>> Max
>>>>> >>>>
>>>>> >>>> On Fri, Oct 10, 2014 at 1:34 PM, Stephan Ewen <se...@apache.org>
>>>>> >>>> wrote:
>>>>> >>>>>
>>>>> >>>>> Thank you, I will look into that...
>>>>> >>>>
>>>>> >>>>
>>>>> >>>
>>>>> >>
>>>>> >
>>>>
>>>>
>>>
>>
>
Re: Forced to use Solution Set in Step Function
Posted by Stephan Ewen <se...@apache.org>.
That is an interesting case. Everything that is loop invariant is computed
once outside the loop. You are looking for a way to make this part of the
loop.
Can you try making the filter part of the "VectorDataSet.add(old_sum,
current)" operation?
On Tue, Oct 14, 2014 at 5:05 PM, Maximilian Alber <
alber.maximilian@gmail.com> wrote:
> The deltaiteration calculates the cumulative sum (prefix sum) of
> residual_2.
>
> I'm not sure if got you idea. I could add residual_2 to the workset. But
> in that case I need to separate the "old_sum"(the sum up to now) and
> residual_2 each iteration. Actually I had that before, then I tried the
> Scala closure to clean up the code.
>
> On Tue, Oct 14, 2014 at 4:57 PM, Fabian Hueske <fh...@apache.org> wrote:
>
>> Jep, I see you point.
>> Conceptually, all data that changes and affects the result of an
>> iteration should be part of the workset.
>> Hence, the model kind of assumes that the datum "superStepNumber" should
>> be part of the workset.
>>
>> I am not familiar with your application, but would it make sense to add
>> the number as an additional attribute to the workset data set and increase
>> it manually?
>>
>> 2014-10-14 16:45 GMT+02:00 Maximilian Alber <al...@gmail.com>:
>>
>>> Ok, sounds true, but somehow I would like to execute it inside of it. So
>>> I probably need to do some nonsense work to make it part of it?
>>>
>>> On Tue, Oct 14, 2014 at 4:36 PM, Aljoscha Krettek <al...@apache.org>
>>> wrote:
>>>
>>>> Dammit you beat me to it. But yes, this is exactly what I was just
>>>> writing.
>>>>
>>>> On Tue, Oct 14, 2014 at 4:35 PM, Fabian Hueske <fh...@apache.org>
>>>> wrote:
>>>> > Hi,
>>>> >
>>>> > I'm not super familiar with the iterations, but my guess would be
>>>> that the
>>>> > filter is not evaluated as part of the iteration.
>>>> > Since it is not connect to the workset, the filter is not part of the
>>>> loop
>>>> > and evaluated once outside where no superset number is available.
>>>> > I guess, moving the filter outside of the loop gives the same error.
>>>> >
>>>> > Cheers, Fabian
>>>> >
>>>> >
>>>> >
>>>> > 2014-10-14 16:18 GMT+02:00 Maximilian Alber <
>>>> alber.maximilian@gmail.com>:
>>>> >>
>>>> >> Hmm or maybe not. With this code I get some strange error:
>>>> >>
>>>> >> def createPlan_find_center(env: ExecutionEnvironment) = {
>>>> >> val X = env readTextFile config.xFile map
>>>> >> {Vector.parseFromString(config.dimensions, _)};
>>>> >> val residual = env readTextFile config.yFile map
>>>> >> {Vector.parseFromString(_)};
>>>> >> val randoms = env readTextFile config.randomFile map
>>>> >> {Vector.parseFromString(_)}
>>>> >>
>>>> >> val residual_2 = residual * residual
>>>> >> val ys = (residual_2 sumV) * (randoms filter {_.id == 0})
>>>> >>
>>>> >> val emptyDataSet = env.fromCollection[Vector](Seq())
>>>> >> val sumVector =
>>>> env.fromCollection(Seq(Vector.zeros(config.dimensions)))
>>>> >> val cumSum = emptyDataSet.iterateDelta(sumVector, config.N,
>>>> Array("id")) {
>>>> >> (solutionset, old_sum) =>
>>>> >> val current = residual_2 filter (new RichFilterFunction[Vector]{
>>>> >> def filter(x: Vector) = x.id ==
>>>> >> (getIterationRuntimeContext.getSuperstepNumber)
>>>> >> })
>>>> >> val sum = VectorDataSet.add(old_sum, current)
>>>> >>
>>>> >> (sum map (new RichMapFunction[Vector, Vector]{
>>>> >> def map(x: Vector) = new
>>>> >> Vector(getIterationRuntimeContext.getSuperstepNumber, x.values)
>>>> >> }),
>>>> >> sum)
>>>> >> }
>>>> >>
>>>> >> Error:
>>>> >> 10/14/2014 15:57:35: Job execution switched to status RUNNING
>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to
>>>> SCHEDULED
>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to
>>>> DEPLOYING
>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to SCHEDULED
>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to DEPLOYING
>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>>>> (/tmp/tmpBhOsLd) -
>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>>> switched to
>>>> >> SCHEDULED
>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>>>> (/tmp/tmpBhOsLd) -
>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>>> switched to
>>>> >> DEPLOYING
>>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to RUNNING
>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
>>>> >> Iteration)) (1/1) switched to SCHEDULED
>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
>>>> >> Iteration)) (1/1) switched to DEPLOYING
>>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>>>> (/tmp/tmpBhOsLd) -
>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>>> switched to
>>>> >> RUNNING
>>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to RUNNING
>>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
>>>> >> Iteration)) (1/1) switched to RUNNING
>>>> >> 10/14/2014 15:57:35: CHAIN
>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>> Map
>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to SCHEDULED
>>>> >> 10/14/2014 15:57:35: CHAIN
>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>> Map
>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to DEPLOYING
>>>> >> 10/14/2014 15:57:36: CHAIN
>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>> Map
>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>>>> SCHEDULED
>>>> >> 10/14/2014 15:57:36: CHAIN
>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>> Map
>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>>>> DEPLOYING
>>>> >> 10/14/2014 15:57:36: CHAIN
>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>> Map
>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to RUNNING
>>>> >> 10/14/2014 15:57:36: CHAIN
>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>> Map
>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>>>> RUNNING
>>>> >> 10/14/2014 15:57:36: CHAIN
>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>> Map
>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to FAILED
>>>> >> java.lang.IllegalStateException: This stub is not part of an
>>>> iteration
>>>> >> step function.
>>>> >> at
>>>> >>
>>>> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
>>>> >> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
>>>> >> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
>>>> >> at
>>>> >>
>>>> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
>>>> >> at
>>>> >>
>>>> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
>>>> >> at
>>>> >>
>>>> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
>>>> >> at
>>>> >>
>>>> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
>>>> >> at
>>>> >>
>>>> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
>>>> >> at
>>>> >>
>>>> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
>>>> >> at
>>>> >>
>>>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
>>>> >> at
>>>> >>
>>>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
>>>> >> at
>>>> >>
>>>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
>>>> >> at java.lang.Thread.run(Thread.java:745)
>>>> >>
>>>> >> 10/14/2014 15:57:36: Job execution switched to status FAILING
>>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to CANCELING
>>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
>>>> (/tmp/tmpBhOsLd) -
>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>>> switched to
>>>> >> CANCELING
>>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to
>>>> CANCELING
>>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed Delta
>>>> >> Iteration)) (1/1) switched to CANCELING
>>>> >> 10/14/2014 15:57:36: Map (org.apache.flink.api.scala.DataSet$$anon$1)
>>>> >> (1/1) switched to CANCELED
>>>> >> 10/14/2014 15:57:36: DataSink(TextOutputFormat (/tmp/tmplSYJ7S) -
>>>> UTF-8)
>>>> >> (1/1) switched to CANCELED
>>>> >> 10/14/2014 15:57:36: Sync (WorksetIteration (Unnamed Delta
>>>> Iteration))
>>>> >> (1/1) switched to CANCELED
>>>> >> 10/14/2014 15:57:36: CHAIN
>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>> Map
>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>>>> CANCELING
>>>> >> 10/14/2014 15:57:36: Map (bumpboost.BumpBoost$$anonfun$8$$anon$2)
>>>> (1/1)
>>>> >> switched to CANCELED
>>>> >> 10/14/2014 15:57:36: SolutionSet Delta (1/1) switched to CANCELED
>>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to CANCELED
>>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
>>>> (/tmp/tmpBhOsLd) -
>>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>>> switched to
>>>> >> CANCELED
>>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to CANCELED
>>>> >> 10/14/2014 15:57:36: CHAIN
>>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>>> Map
>>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>>>> CANCELED
>>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed Delta
>>>> >> Iteration)) (1/1) switched to CANCELED
>>>> >> 10/14/2014 15:57:36: Job execution switched to status FAILED
>>>> >> Error: The program execution failed:
>>>> java.lang.IllegalStateException: This
>>>> >> stub is not part of an iteration step function.
>>>> >> at
>>>> >>
>>>> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
>>>> >> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
>>>> >> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
>>>> >> at
>>>> >>
>>>> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
>>>> >> at
>>>> >>
>>>> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
>>>> >> at
>>>> >>
>>>> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
>>>> >> at
>>>> >>
>>>> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
>>>> >> at
>>>> >>
>>>> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
>>>> >> at
>>>> >>
>>>> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
>>>> >> at
>>>> >>
>>>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
>>>> >> at
>>>> >>
>>>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
>>>> >> at
>>>> >>
>>>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
>>>> >> at java.lang.Thread.run(Thread.java:745)
>>>> >>
>>>> >> On Tue, Oct 14, 2014 at 3:32 PM, Maximilian Alber
>>>> >> <al...@gmail.com> wrote:
>>>> >>>
>>>> >>> Should work now.
>>>> >>> Cheers
>>>> >>>
>>>> >>> On Fri, Oct 10, 2014 at 3:38 PM, Maximilian Alber
>>>> >>> <al...@gmail.com> wrote:
>>>> >>>>
>>>> >>>> Ok, thanks.
>>>> >>>> Please let me know when it is fixed.
>>>> >>>>
>>>> >>>> Cheers
>>>> >>>> Max
>>>> >>>>
>>>> >>>> On Fri, Oct 10, 2014 at 1:34 PM, Stephan Ewen <se...@apache.org>
>>>> wrote:
>>>> >>>>>
>>>> >>>>> Thank you, I will look into that...
>>>> >>>>
>>>> >>>>
>>>> >>>
>>>> >>
>>>> >
>>>>
>>>
>>>
>>
>
Re: Forced to use Solution Set in Step Function
Posted by Maximilian Alber <al...@gmail.com>.
The deltaiteration calculates the cumulative sum (prefix sum) of residual_2.
I'm not sure if got you idea. I could add residual_2 to the workset. But in
that case I need to separate the "old_sum"(the sum up to now) and
residual_2 each iteration. Actually I had that before, then I tried the
Scala closure to clean up the code.
On Tue, Oct 14, 2014 at 4:57 PM, Fabian Hueske <fh...@apache.org> wrote:
> Jep, I see you point.
> Conceptually, all data that changes and affects the result of an iteration
> should be part of the workset.
> Hence, the model kind of assumes that the datum "superStepNumber" should
> be part of the workset.
>
> I am not familiar with your application, but would it make sense to add
> the number as an additional attribute to the workset data set and increase
> it manually?
>
> 2014-10-14 16:45 GMT+02:00 Maximilian Alber <al...@gmail.com>:
>
>> Ok, sounds true, but somehow I would like to execute it inside of it. So
>> I probably need to do some nonsense work to make it part of it?
>>
>> On Tue, Oct 14, 2014 at 4:36 PM, Aljoscha Krettek <al...@apache.org>
>> wrote:
>>
>>> Dammit you beat me to it. But yes, this is exactly what I was just
>>> writing.
>>>
>>> On Tue, Oct 14, 2014 at 4:35 PM, Fabian Hueske <fh...@apache.org>
>>> wrote:
>>> > Hi,
>>> >
>>> > I'm not super familiar with the iterations, but my guess would be that
>>> the
>>> > filter is not evaluated as part of the iteration.
>>> > Since it is not connect to the workset, the filter is not part of the
>>> loop
>>> > and evaluated once outside where no superset number is available.
>>> > I guess, moving the filter outside of the loop gives the same error.
>>> >
>>> > Cheers, Fabian
>>> >
>>> >
>>> >
>>> > 2014-10-14 16:18 GMT+02:00 Maximilian Alber <
>>> alber.maximilian@gmail.com>:
>>> >>
>>> >> Hmm or maybe not. With this code I get some strange error:
>>> >>
>>> >> def createPlan_find_center(env: ExecutionEnvironment) = {
>>> >> val X = env readTextFile config.xFile map
>>> >> {Vector.parseFromString(config.dimensions, _)};
>>> >> val residual = env readTextFile config.yFile map
>>> >> {Vector.parseFromString(_)};
>>> >> val randoms = env readTextFile config.randomFile map
>>> >> {Vector.parseFromString(_)}
>>> >>
>>> >> val residual_2 = residual * residual
>>> >> val ys = (residual_2 sumV) * (randoms filter {_.id == 0})
>>> >>
>>> >> val emptyDataSet = env.fromCollection[Vector](Seq())
>>> >> val sumVector =
>>> env.fromCollection(Seq(Vector.zeros(config.dimensions)))
>>> >> val cumSum = emptyDataSet.iterateDelta(sumVector, config.N,
>>> Array("id")) {
>>> >> (solutionset, old_sum) =>
>>> >> val current = residual_2 filter (new RichFilterFunction[Vector]{
>>> >> def filter(x: Vector) = x.id ==
>>> >> (getIterationRuntimeContext.getSuperstepNumber)
>>> >> })
>>> >> val sum = VectorDataSet.add(old_sum, current)
>>> >>
>>> >> (sum map (new RichMapFunction[Vector, Vector]{
>>> >> def map(x: Vector) = new
>>> >> Vector(getIterationRuntimeContext.getSuperstepNumber, x.values)
>>> >> }),
>>> >> sum)
>>> >> }
>>> >>
>>> >> Error:
>>> >> 10/14/2014 15:57:35: Job execution switched to status RUNNING
>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to SCHEDULED
>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to DEPLOYING
>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to SCHEDULED
>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to DEPLOYING
>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>>> (/tmp/tmpBhOsLd) -
>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>> switched to
>>> >> SCHEDULED
>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>>> (/tmp/tmpBhOsLd) -
>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>> switched to
>>> >> DEPLOYING
>>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to RUNNING
>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
>>> >> Iteration)) (1/1) switched to SCHEDULED
>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
>>> >> Iteration)) (1/1) switched to DEPLOYING
>>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>>> (/tmp/tmpBhOsLd) -
>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>> switched to
>>> >> RUNNING
>>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to RUNNING
>>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
>>> >> Iteration)) (1/1) switched to RUNNING
>>> >> 10/14/2014 15:57:35: CHAIN
>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>> Map
>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to SCHEDULED
>>> >> 10/14/2014 15:57:35: CHAIN
>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>> Map
>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to DEPLOYING
>>> >> 10/14/2014 15:57:36: CHAIN
>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>> Map
>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>>> SCHEDULED
>>> >> 10/14/2014 15:57:36: CHAIN
>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>> Map
>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>>> DEPLOYING
>>> >> 10/14/2014 15:57:36: CHAIN
>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>> Map
>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to RUNNING
>>> >> 10/14/2014 15:57:36: CHAIN
>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>> Map
>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to RUNNING
>>> >> 10/14/2014 15:57:36: CHAIN
>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>> Map
>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to FAILED
>>> >> java.lang.IllegalStateException: This stub is not part of an iteration
>>> >> step function.
>>> >> at
>>> >>
>>> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
>>> >> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
>>> >> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
>>> >> at
>>> >>
>>> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
>>> >> at
>>> >>
>>> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
>>> >> at
>>> >>
>>> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
>>> >> at
>>> >>
>>> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
>>> >> at
>>> >>
>>> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
>>> >> at
>>> >>
>>> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
>>> >> at
>>> >>
>>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
>>> >> at
>>> >>
>>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
>>> >> at
>>> >>
>>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
>>> >> at java.lang.Thread.run(Thread.java:745)
>>> >>
>>> >> 10/14/2014 15:57:36: Job execution switched to status FAILING
>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to CANCELING
>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
>>> (/tmp/tmpBhOsLd) -
>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>> switched to
>>> >> CANCELING
>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to CANCELING
>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed Delta
>>> >> Iteration)) (1/1) switched to CANCELING
>>> >> 10/14/2014 15:57:36: Map (org.apache.flink.api.scala.DataSet$$anon$1)
>>> >> (1/1) switched to CANCELED
>>> >> 10/14/2014 15:57:36: DataSink(TextOutputFormat (/tmp/tmplSYJ7S) -
>>> UTF-8)
>>> >> (1/1) switched to CANCELED
>>> >> 10/14/2014 15:57:36: Sync (WorksetIteration (Unnamed Delta Iteration))
>>> >> (1/1) switched to CANCELED
>>> >> 10/14/2014 15:57:36: CHAIN
>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>> Map
>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>>> CANCELING
>>> >> 10/14/2014 15:57:36: Map (bumpboost.BumpBoost$$anonfun$8$$anon$2)
>>> (1/1)
>>> >> switched to CANCELED
>>> >> 10/14/2014 15:57:36: SolutionSet Delta (1/1) switched to CANCELED
>>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to CANCELED
>>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
>>> (/tmp/tmpBhOsLd) -
>>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>>> switched to
>>> >> CANCELED
>>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to CANCELED
>>> >> 10/14/2014 15:57:36: CHAIN
>>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) ->
>>> Map
>>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>>> CANCELED
>>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed Delta
>>> >> Iteration)) (1/1) switched to CANCELED
>>> >> 10/14/2014 15:57:36: Job execution switched to status FAILED
>>> >> Error: The program execution failed: java.lang.IllegalStateException:
>>> This
>>> >> stub is not part of an iteration step function.
>>> >> at
>>> >>
>>> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
>>> >> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
>>> >> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
>>> >> at
>>> >>
>>> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
>>> >> at
>>> >>
>>> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
>>> >> at
>>> >>
>>> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
>>> >> at
>>> >>
>>> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
>>> >> at
>>> >>
>>> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
>>> >> at
>>> >>
>>> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
>>> >> at
>>> >>
>>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
>>> >> at
>>> >>
>>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
>>> >> at
>>> >>
>>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
>>> >> at java.lang.Thread.run(Thread.java:745)
>>> >>
>>> >> On Tue, Oct 14, 2014 at 3:32 PM, Maximilian Alber
>>> >> <al...@gmail.com> wrote:
>>> >>>
>>> >>> Should work now.
>>> >>> Cheers
>>> >>>
>>> >>> On Fri, Oct 10, 2014 at 3:38 PM, Maximilian Alber
>>> >>> <al...@gmail.com> wrote:
>>> >>>>
>>> >>>> Ok, thanks.
>>> >>>> Please let me know when it is fixed.
>>> >>>>
>>> >>>> Cheers
>>> >>>> Max
>>> >>>>
>>> >>>> On Fri, Oct 10, 2014 at 1:34 PM, Stephan Ewen <se...@apache.org>
>>> wrote:
>>> >>>>>
>>> >>>>> Thank you, I will look into that...
>>> >>>>
>>> >>>>
>>> >>>
>>> >>
>>> >
>>>
>>
>>
>
Re: Forced to use Solution Set in Step Function
Posted by Fabian Hueske <fh...@apache.org>.
Jep, I see you point.
Conceptually, all data that changes and affects the result of an iteration
should be part of the workset.
Hence, the model kind of assumes that the datum "superStepNumber" should be
part of the workset.
I am not familiar with your application, but would it make sense to add the
number as an additional attribute to the workset data set and increase it
manually?
2014-10-14 16:45 GMT+02:00 Maximilian Alber <al...@gmail.com>:
> Ok, sounds true, but somehow I would like to execute it inside of it. So I
> probably need to do some nonsense work to make it part of it?
>
> On Tue, Oct 14, 2014 at 4:36 PM, Aljoscha Krettek <al...@apache.org>
> wrote:
>
>> Dammit you beat me to it. But yes, this is exactly what I was just
>> writing.
>>
>> On Tue, Oct 14, 2014 at 4:35 PM, Fabian Hueske <fh...@apache.org>
>> wrote:
>> > Hi,
>> >
>> > I'm not super familiar with the iterations, but my guess would be that
>> the
>> > filter is not evaluated as part of the iteration.
>> > Since it is not connect to the workset, the filter is not part of the
>> loop
>> > and evaluated once outside where no superset number is available.
>> > I guess, moving the filter outside of the loop gives the same error.
>> >
>> > Cheers, Fabian
>> >
>> >
>> >
>> > 2014-10-14 16:18 GMT+02:00 Maximilian Alber <alber.maximilian@gmail.com
>> >:
>> >>
>> >> Hmm or maybe not. With this code I get some strange error:
>> >>
>> >> def createPlan_find_center(env: ExecutionEnvironment) = {
>> >> val X = env readTextFile config.xFile map
>> >> {Vector.parseFromString(config.dimensions, _)};
>> >> val residual = env readTextFile config.yFile map
>> >> {Vector.parseFromString(_)};
>> >> val randoms = env readTextFile config.randomFile map
>> >> {Vector.parseFromString(_)}
>> >>
>> >> val residual_2 = residual * residual
>> >> val ys = (residual_2 sumV) * (randoms filter {_.id == 0})
>> >>
>> >> val emptyDataSet = env.fromCollection[Vector](Seq())
>> >> val sumVector =
>> env.fromCollection(Seq(Vector.zeros(config.dimensions)))
>> >> val cumSum = emptyDataSet.iterateDelta(sumVector, config.N,
>> Array("id")) {
>> >> (solutionset, old_sum) =>
>> >> val current = residual_2 filter (new RichFilterFunction[Vector]{
>> >> def filter(x: Vector) = x.id ==
>> >> (getIterationRuntimeContext.getSuperstepNumber)
>> >> })
>> >> val sum = VectorDataSet.add(old_sum, current)
>> >>
>> >> (sum map (new RichMapFunction[Vector, Vector]{
>> >> def map(x: Vector) = new
>> >> Vector(getIterationRuntimeContext.getSuperstepNumber, x.values)
>> >> }),
>> >> sum)
>> >> }
>> >>
>> >> Error:
>> >> 10/14/2014 15:57:35: Job execution switched to status RUNNING
>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to SCHEDULED
>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to DEPLOYING
>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to SCHEDULED
>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to DEPLOYING
>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>> (/tmp/tmpBhOsLd) -
>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>> switched to
>> >> SCHEDULED
>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>> (/tmp/tmpBhOsLd) -
>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>> switched to
>> >> DEPLOYING
>> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to RUNNING
>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
>> >> Iteration)) (1/1) switched to SCHEDULED
>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
>> >> Iteration)) (1/1) switched to DEPLOYING
>> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat
>> (/tmp/tmpBhOsLd) -
>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>> switched to
>> >> RUNNING
>> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to RUNNING
>> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
>> >> Iteration)) (1/1) switched to RUNNING
>> >> 10/14/2014 15:57:35: CHAIN
>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to SCHEDULED
>> >> 10/14/2014 15:57:35: CHAIN
>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to DEPLOYING
>> >> 10/14/2014 15:57:36: CHAIN
>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>> SCHEDULED
>> >> 10/14/2014 15:57:36: CHAIN
>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>> DEPLOYING
>> >> 10/14/2014 15:57:36: CHAIN
>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to RUNNING
>> >> 10/14/2014 15:57:36: CHAIN
>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to RUNNING
>> >> 10/14/2014 15:57:36: CHAIN
>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
>> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to FAILED
>> >> java.lang.IllegalStateException: This stub is not part of an iteration
>> >> step function.
>> >> at
>> >>
>> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
>> >> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
>> >> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
>> >> at
>> >>
>> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
>> >> at
>> >>
>> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
>> >> at
>> >>
>> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
>> >> at
>> >>
>> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
>> >> at
>> >>
>> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
>> >> at
>> >>
>> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
>> >> at
>> >>
>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
>> >> at
>> >>
>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
>> >> at
>> >>
>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
>> >> at java.lang.Thread.run(Thread.java:745)
>> >>
>> >> 10/14/2014 15:57:36: Job execution switched to status FAILING
>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to CANCELING
>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
>> (/tmp/tmpBhOsLd) -
>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>> switched to
>> >> CANCELING
>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to CANCELING
>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed Delta
>> >> Iteration)) (1/1) switched to CANCELING
>> >> 10/14/2014 15:57:36: Map (org.apache.flink.api.scala.DataSet$$anon$1)
>> >> (1/1) switched to CANCELED
>> >> 10/14/2014 15:57:36: DataSink(TextOutputFormat (/tmp/tmplSYJ7S) -
>> UTF-8)
>> >> (1/1) switched to CANCELED
>> >> 10/14/2014 15:57:36: Sync (WorksetIteration (Unnamed Delta Iteration))
>> >> (1/1) switched to CANCELED
>> >> 10/14/2014 15:57:36: CHAIN
>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>> CANCELING
>> >> 10/14/2014 15:57:36: Map (bumpboost.BumpBoost$$anonfun$8$$anon$2) (1/1)
>> >> switched to CANCELED
>> >> 10/14/2014 15:57:36: SolutionSet Delta (1/1) switched to CANCELED
>> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to CANCELED
>> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat
>> (/tmp/tmpBhOsLd) -
>> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
>> switched to
>> >> CANCELED
>> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to CANCELED
>> >> 10/14/2014 15:57:36: CHAIN
>> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
>> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to CANCELED
>> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed Delta
>> >> Iteration)) (1/1) switched to CANCELED
>> >> 10/14/2014 15:57:36: Job execution switched to status FAILED
>> >> Error: The program execution failed: java.lang.IllegalStateException:
>> This
>> >> stub is not part of an iteration step function.
>> >> at
>> >>
>> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
>> >> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
>> >> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
>> >> at
>> >>
>> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
>> >> at
>> >>
>> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
>> >> at
>> >>
>> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
>> >> at
>> >>
>> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
>> >> at
>> >>
>> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
>> >> at
>> >>
>> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
>> >> at
>> >>
>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
>> >> at
>> >>
>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
>> >> at
>> >>
>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
>> >> at java.lang.Thread.run(Thread.java:745)
>> >>
>> >> On Tue, Oct 14, 2014 at 3:32 PM, Maximilian Alber
>> >> <al...@gmail.com> wrote:
>> >>>
>> >>> Should work now.
>> >>> Cheers
>> >>>
>> >>> On Fri, Oct 10, 2014 at 3:38 PM, Maximilian Alber
>> >>> <al...@gmail.com> wrote:
>> >>>>
>> >>>> Ok, thanks.
>> >>>> Please let me know when it is fixed.
>> >>>>
>> >>>> Cheers
>> >>>> Max
>> >>>>
>> >>>> On Fri, Oct 10, 2014 at 1:34 PM, Stephan Ewen <se...@apache.org>
>> wrote:
>> >>>>>
>> >>>>> Thank you, I will look into that...
>> >>>>
>> >>>>
>> >>>
>> >>
>> >
>>
>
>
Re: Forced to use Solution Set in Step Function
Posted by Maximilian Alber <al...@gmail.com>.
Ok, sounds true, but somehow I would like to execute it inside of it. So I
probably need to do some nonsense work to make it part of it?
On Tue, Oct 14, 2014 at 4:36 PM, Aljoscha Krettek <al...@apache.org>
wrote:
> Dammit you beat me to it. But yes, this is exactly what I was just writing.
>
> On Tue, Oct 14, 2014 at 4:35 PM, Fabian Hueske <fh...@apache.org> wrote:
> > Hi,
> >
> > I'm not super familiar with the iterations, but my guess would be that
> the
> > filter is not evaluated as part of the iteration.
> > Since it is not connect to the workset, the filter is not part of the
> loop
> > and evaluated once outside where no superset number is available.
> > I guess, moving the filter outside of the loop gives the same error.
> >
> > Cheers, Fabian
> >
> >
> >
> > 2014-10-14 16:18 GMT+02:00 Maximilian Alber <alber.maximilian@gmail.com
> >:
> >>
> >> Hmm or maybe not. With this code I get some strange error:
> >>
> >> def createPlan_find_center(env: ExecutionEnvironment) = {
> >> val X = env readTextFile config.xFile map
> >> {Vector.parseFromString(config.dimensions, _)};
> >> val residual = env readTextFile config.yFile map
> >> {Vector.parseFromString(_)};
> >> val randoms = env readTextFile config.randomFile map
> >> {Vector.parseFromString(_)}
> >>
> >> val residual_2 = residual * residual
> >> val ys = (residual_2 sumV) * (randoms filter {_.id == 0})
> >>
> >> val emptyDataSet = env.fromCollection[Vector](Seq())
> >> val sumVector = env.fromCollection(Seq(Vector.zeros(config.dimensions)))
> >> val cumSum = emptyDataSet.iterateDelta(sumVector, config.N,
> Array("id")) {
> >> (solutionset, old_sum) =>
> >> val current = residual_2 filter (new RichFilterFunction[Vector]{
> >> def filter(x: Vector) = x.id ==
> >> (getIterationRuntimeContext.getSuperstepNumber)
> >> })
> >> val sum = VectorDataSet.add(old_sum, current)
> >>
> >> (sum map (new RichMapFunction[Vector, Vector]{
> >> def map(x: Vector) = new
> >> Vector(getIterationRuntimeContext.getSuperstepNumber, x.values)
> >> }),
> >> sum)
> >> }
> >>
> >> Error:
> >> 10/14/2014 15:57:35: Job execution switched to status RUNNING
> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to SCHEDULED
> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to DEPLOYING
> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to SCHEDULED
> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to DEPLOYING
> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat (/tmp/tmpBhOsLd)
> -
> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
> switched to
> >> SCHEDULED
> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat (/tmp/tmpBhOsLd)
> -
> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
> switched to
> >> DEPLOYING
> >> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to RUNNING
> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
> >> Iteration)) (1/1) switched to SCHEDULED
> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
> >> Iteration)) (1/1) switched to DEPLOYING
> >> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat (/tmp/tmpBhOsLd)
> -
> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
> switched to
> >> RUNNING
> >> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to RUNNING
> >> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
> >> Iteration)) (1/1) switched to RUNNING
> >> 10/14/2014 15:57:35: CHAIN
> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to SCHEDULED
> >> 10/14/2014 15:57:35: CHAIN
> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to DEPLOYING
> >> 10/14/2014 15:57:36: CHAIN
> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to SCHEDULED
> >> 10/14/2014 15:57:36: CHAIN
> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to DEPLOYING
> >> 10/14/2014 15:57:36: CHAIN
> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to RUNNING
> >> 10/14/2014 15:57:36: CHAIN
> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to RUNNING
> >> 10/14/2014 15:57:36: CHAIN
> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
> >> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
> >> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to FAILED
> >> java.lang.IllegalStateException: This stub is not part of an iteration
> >> step function.
> >> at
> >>
> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
> >> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
> >> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
> >> at
> >>
> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
> >> at
> >>
> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
> >> at
> >>
> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
> >> at
> >>
> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
> >> at
> >>
> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
> >> at
> >> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
> >> at
> >>
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
> >> at
> >>
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
> >> at
> >>
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
> >> at java.lang.Thread.run(Thread.java:745)
> >>
> >> 10/14/2014 15:57:36: Job execution switched to status FAILING
> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to CANCELING
> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat (/tmp/tmpBhOsLd)
> -
> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
> switched to
> >> CANCELING
> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to CANCELING
> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed Delta
> >> Iteration)) (1/1) switched to CANCELING
> >> 10/14/2014 15:57:36: Map (org.apache.flink.api.scala.DataSet$$anon$1)
> >> (1/1) switched to CANCELED
> >> 10/14/2014 15:57:36: DataSink(TextOutputFormat (/tmp/tmplSYJ7S) - UTF-8)
> >> (1/1) switched to CANCELED
> >> 10/14/2014 15:57:36: Sync (WorksetIteration (Unnamed Delta Iteration))
> >> (1/1) switched to CANCELED
> >> 10/14/2014 15:57:36: CHAIN
> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to CANCELING
> >> 10/14/2014 15:57:36: Map (bumpboost.BumpBoost$$anonfun$8$$anon$2) (1/1)
> >> switched to CANCELED
> >> 10/14/2014 15:57:36: SolutionSet Delta (1/1) switched to CANCELED
> >> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to CANCELED
> >> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat (/tmp/tmpBhOsLd)
> -
> >> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
> switched to
> >> CANCELED
> >> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to CANCELED
> >> 10/14/2014 15:57:36: CHAIN
> >> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
> >> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to CANCELED
> >> 10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed Delta
> >> Iteration)) (1/1) switched to CANCELED
> >> 10/14/2014 15:57:36: Job execution switched to status FAILED
> >> Error: The program execution failed: java.lang.IllegalStateException:
> This
> >> stub is not part of an iteration step function.
> >> at
> >>
> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
> >> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
> >> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
> >> at
> >>
> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
> >> at
> >>
> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
> >> at
> >>
> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
> >> at
> >>
> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
> >> at
> >>
> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
> >> at
> >> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
> >> at
> >>
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
> >> at
> >>
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
> >> at
> >>
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
> >> at java.lang.Thread.run(Thread.java:745)
> >>
> >> On Tue, Oct 14, 2014 at 3:32 PM, Maximilian Alber
> >> <al...@gmail.com> wrote:
> >>>
> >>> Should work now.
> >>> Cheers
> >>>
> >>> On Fri, Oct 10, 2014 at 3:38 PM, Maximilian Alber
> >>> <al...@gmail.com> wrote:
> >>>>
> >>>> Ok, thanks.
> >>>> Please let me know when it is fixed.
> >>>>
> >>>> Cheers
> >>>> Max
> >>>>
> >>>> On Fri, Oct 10, 2014 at 1:34 PM, Stephan Ewen <se...@apache.org>
> wrote:
> >>>>>
> >>>>> Thank you, I will look into that...
> >>>>
> >>>>
> >>>
> >>
> >
>
Re: Forced to use Solution Set in Step Function
Posted by Aljoscha Krettek <al...@apache.org>.
Dammit you beat me to it. But yes, this is exactly what I was just writing.
On Tue, Oct 14, 2014 at 4:35 PM, Fabian Hueske <fh...@apache.org> wrote:
> Hi,
>
> I'm not super familiar with the iterations, but my guess would be that the
> filter is not evaluated as part of the iteration.
> Since it is not connect to the workset, the filter is not part of the loop
> and evaluated once outside where no superset number is available.
> I guess, moving the filter outside of the loop gives the same error.
>
> Cheers, Fabian
>
>
>
> 2014-10-14 16:18 GMT+02:00 Maximilian Alber <al...@gmail.com>:
>>
>> Hmm or maybe not. With this code I get some strange error:
>>
>> def createPlan_find_center(env: ExecutionEnvironment) = {
>> val X = env readTextFile config.xFile map
>> {Vector.parseFromString(config.dimensions, _)};
>> val residual = env readTextFile config.yFile map
>> {Vector.parseFromString(_)};
>> val randoms = env readTextFile config.randomFile map
>> {Vector.parseFromString(_)}
>>
>> val residual_2 = residual * residual
>> val ys = (residual_2 sumV) * (randoms filter {_.id == 0})
>>
>> val emptyDataSet = env.fromCollection[Vector](Seq())
>> val sumVector = env.fromCollection(Seq(Vector.zeros(config.dimensions)))
>> val cumSum = emptyDataSet.iterateDelta(sumVector, config.N, Array("id")) {
>> (solutionset, old_sum) =>
>> val current = residual_2 filter (new RichFilterFunction[Vector]{
>> def filter(x: Vector) = x.id ==
>> (getIterationRuntimeContext.getSuperstepNumber)
>> })
>> val sum = VectorDataSet.add(old_sum, current)
>>
>> (sum map (new RichMapFunction[Vector, Vector]{
>> def map(x: Vector) = new
>> Vector(getIterationRuntimeContext.getSuperstepNumber, x.values)
>> }),
>> sum)
>> }
>>
>> Error:
>> 10/14/2014 15:57:35: Job execution switched to status RUNNING
>> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to SCHEDULED
>> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to DEPLOYING
>> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to SCHEDULED
>> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to DEPLOYING
>> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat (/tmp/tmpBhOsLd) -
>> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>> SCHEDULED
>> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat (/tmp/tmpBhOsLd) -
>> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>> DEPLOYING
>> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to RUNNING
>> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
>> Iteration)) (1/1) switched to SCHEDULED
>> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
>> Iteration)) (1/1) switched to DEPLOYING
>> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat (/tmp/tmpBhOsLd) -
>> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>> RUNNING
>> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to RUNNING
>> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
>> Iteration)) (1/1) switched to RUNNING
>> 10/14/2014 15:57:35: CHAIN
>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
>> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to SCHEDULED
>> 10/14/2014 15:57:35: CHAIN
>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
>> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to DEPLOYING
>> 10/14/2014 15:57:36: CHAIN
>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
>> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to SCHEDULED
>> 10/14/2014 15:57:36: CHAIN
>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
>> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to DEPLOYING
>> 10/14/2014 15:57:36: CHAIN
>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
>> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to RUNNING
>> 10/14/2014 15:57:36: CHAIN
>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
>> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to RUNNING
>> 10/14/2014 15:57:36: CHAIN
>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
>> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
>> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to FAILED
>> java.lang.IllegalStateException: This stub is not part of an iteration
>> step function.
>> at
>> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
>> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
>> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
>> at
>> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
>> at
>> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
>> at
>> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
>> at
>> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
>> at
>> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
>> at
>> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
>> at
>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
>> at
>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
>> at
>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> 10/14/2014 15:57:36: Job execution switched to status FAILING
>> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to CANCELING
>> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat (/tmp/tmpBhOsLd) -
>> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>> CANCELING
>> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to CANCELING
>> 10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed Delta
>> Iteration)) (1/1) switched to CANCELING
>> 10/14/2014 15:57:36: Map (org.apache.flink.api.scala.DataSet$$anon$1)
>> (1/1) switched to CANCELED
>> 10/14/2014 15:57:36: DataSink(TextOutputFormat (/tmp/tmplSYJ7S) - UTF-8)
>> (1/1) switched to CANCELED
>> 10/14/2014 15:57:36: Sync (WorksetIteration (Unnamed Delta Iteration))
>> (1/1) switched to CANCELED
>> 10/14/2014 15:57:36: CHAIN
>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
>> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to CANCELING
>> 10/14/2014 15:57:36: Map (bumpboost.BumpBoost$$anonfun$8$$anon$2) (1/1)
>> switched to CANCELED
>> 10/14/2014 15:57:36: SolutionSet Delta (1/1) switched to CANCELED
>> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to CANCELED
>> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat (/tmp/tmpBhOsLd) -
>> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to
>> CANCELED
>> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to CANCELED
>> 10/14/2014 15:57:36: CHAIN
>> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
>> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to CANCELED
>> 10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed Delta
>> Iteration)) (1/1) switched to CANCELED
>> 10/14/2014 15:57:36: Job execution switched to status FAILED
>> Error: The program execution failed: java.lang.IllegalStateException: This
>> stub is not part of an iteration step function.
>> at
>> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
>> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
>> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
>> at
>> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
>> at
>> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
>> at
>> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
>> at
>> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
>> at
>> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
>> at
>> org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
>> at
>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
>> at
>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
>> at
>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> On Tue, Oct 14, 2014 at 3:32 PM, Maximilian Alber
>> <al...@gmail.com> wrote:
>>>
>>> Should work now.
>>> Cheers
>>>
>>> On Fri, Oct 10, 2014 at 3:38 PM, Maximilian Alber
>>> <al...@gmail.com> wrote:
>>>>
>>>> Ok, thanks.
>>>> Please let me know when it is fixed.
>>>>
>>>> Cheers
>>>> Max
>>>>
>>>> On Fri, Oct 10, 2014 at 1:34 PM, Stephan Ewen <se...@apache.org> wrote:
>>>>>
>>>>> Thank you, I will look into that...
>>>>
>>>>
>>>
>>
>
Re: Forced to use Solution Set in Step Function
Posted by Fabian Hueske <fh...@apache.org>.
Hi,
I'm not super familiar with the iterations, but my guess would be that the
filter is not evaluated as part of the iteration.
Since it is not connect to the workset, the filter is not part of the loop
and evaluated once outside where no superset number is available.
I guess, moving the filter outside of the loop gives the same error.
Cheers, Fabian
2014-10-14 16:18 GMT+02:00 Maximilian Alber <al...@gmail.com>:
> Hmm or maybe not. With this code I get some strange error:
>
> def createPlan_find_center(env: ExecutionEnvironment) = {
> val X = env readTextFile config.xFile map
> {Vector.parseFromString(config.dimensions, _)};
> val residual = env readTextFile config.yFile map
> {Vector.parseFromString(_)};
> val randoms = env readTextFile config.randomFile map
> {Vector.parseFromString(_)}
>
> val residual_2 = residual * residual
> val ys = (residual_2 sumV) * (randoms filter {_.id == 0})
>
> val emptyDataSet = env.fromCollection[Vector](Seq())
> val sumVector = env.fromCollection(Seq(Vector.zeros(config.dimensions)))
> val cumSum = emptyDataSet.iterateDelta(sumVector, config.N, Array("id")) {
> (solutionset, old_sum) =>
> val current = residual_2 filter (new RichFilterFunction[Vector]{
> def filter(x: Vector) = x.id ==
> (getIterationRuntimeContext.getSuperstepNumber)
> })
> val sum = VectorDataSet.add(old_sum, current)
>
> (sum map (new RichMapFunction[Vector, Vector]{
> def map(x: Vector) = new
> Vector(getIterationRuntimeContext.getSuperstepNumber, x.values)
> }),
> sum)
> }
>
> Error:
> 10/14/2014 15:57:35: Job execution switched to status RUNNING
> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to SCHEDULED
> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to DEPLOYING
> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to SCHEDULED
> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to DEPLOYING
> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat (/tmp/tmpBhOsLd) -
> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched
> to SCHEDULED
> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat (/tmp/tmpBhOsLd) -
> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched
> to DEPLOYING
> 10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to RUNNING
> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
> Iteration)) (1/1) switched to SCHEDULED
> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
> Iteration)) (1/1) switched to DEPLOYING
> 10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat (/tmp/tmpBhOsLd) -
> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched
> to RUNNING
> 10/14/2014 15:57:35: DataSource ([]) (1/1) switched to RUNNING
> 10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
> Iteration)) (1/1) switched to RUNNING
> 10/14/2014 15:57:35: CHAIN
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to SCHEDULED
> 10/14/2014 15:57:35: CHAIN
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to DEPLOYING
> 10/14/2014 15:57:36: CHAIN
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to SCHEDULED
> 10/14/2014 15:57:36: CHAIN
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to DEPLOYING
> 10/14/2014 15:57:36: CHAIN
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to RUNNING
> 10/14/2014 15:57:36: CHAIN
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to RUNNING
> 10/14/2014 15:57:36: CHAIN
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
> (org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
> (bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to FAILED
> java.lang.IllegalStateException: This stub is not part of an iteration
> step function.
> at
> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
> at
> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
> at
> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
> at
> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
> at
> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
> at
> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
> at org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
> at
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
> at
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
> at
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
> at java.lang.Thread.run(Thread.java:745)
>
> 10/14/2014 15:57:36: Job execution switched to status FAILING
> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to CANCELING
> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat (/tmp/tmpBhOsLd) -
> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched
> to CANCELING
> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to CANCELING
> 10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed Delta
> Iteration)) (1/1) switched to CANCELING
> 10/14/2014 15:57:36: Map (org.apache.flink.api.scala.DataSet$$anon$1)
> (1/1) switched to CANCELED
> 10/14/2014 15:57:36: DataSink(TextOutputFormat (/tmp/tmplSYJ7S) - UTF-8)
> (1/1) switched to CANCELED
> 10/14/2014 15:57:36: Sync (WorksetIteration (Unnamed Delta Iteration))
> (1/1) switched to CANCELED
> 10/14/2014 15:57:36: CHAIN
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to CANCELING
> 10/14/2014 15:57:36: Map (bumpboost.BumpBoost$$anonfun$8$$anon$2) (1/1)
> switched to CANCELED
> 10/14/2014 15:57:36: SolutionSet Delta (1/1) switched to CANCELED
> 10/14/2014 15:57:36: DataSource ([]) (1/1) switched to CANCELED
> 10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat (/tmp/tmpBhOsLd) -
> UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched
> to CANCELED
> 10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to CANCELED
> 10/14/2014 15:57:36: CHAIN
> Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
> (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to CANCELED
> 10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed Delta
> Iteration)) (1/1) switched to CANCELED
> 10/14/2014 15:57:36: Job execution switched to status FAILED
> Error: The program execution failed: java.lang.IllegalStateException: This
> stub is not part of an iteration step function.
> at
> org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
> at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
> at
> org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
> at
> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
> at
> org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
> at
> org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
> at
> org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
> at org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
> at
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
> at
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
> at
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
> at java.lang.Thread.run(Thread.java:745)
>
> On Tue, Oct 14, 2014 at 3:32 PM, Maximilian Alber <
> alber.maximilian@gmail.com> wrote:
>
>> Should work now.
>> Cheers
>>
>> On Fri, Oct 10, 2014 at 3:38 PM, Maximilian Alber <
>> alber.maximilian@gmail.com> wrote:
>>
>>> Ok, thanks.
>>> Please let me know when it is fixed.
>>>
>>> Cheers
>>> Max
>>>
>>> On Fri, Oct 10, 2014 at 1:34 PM, Stephan Ewen <se...@apache.org> wrote:
>>>
>>>> Thank you, I will look into that...
>>>>
>>>
>>>
>>
>
Re: Forced to use Solution Set in Step Function
Posted by Maximilian Alber <al...@gmail.com>.
Hmm or maybe not. With this code I get some strange error:
def createPlan_find_center(env: ExecutionEnvironment) = {
val X = env readTextFile config.xFile map
{Vector.parseFromString(config.dimensions, _)};
val residual = env readTextFile config.yFile map
{Vector.parseFromString(_)};
val randoms = env readTextFile config.randomFile map
{Vector.parseFromString(_)}
val residual_2 = residual * residual
val ys = (residual_2 sumV) * (randoms filter {_.id == 0})
val emptyDataSet = env.fromCollection[Vector](Seq())
val sumVector = env.fromCollection(Seq(Vector.zeros(config.dimensions)))
val cumSum = emptyDataSet.iterateDelta(sumVector, config.N, Array("id")) {
(solutionset, old_sum) =>
val current = residual_2 filter (new RichFilterFunction[Vector]{
def filter(x: Vector) = x.id ==
(getIterationRuntimeContext.getSuperstepNumber)
})
val sum = VectorDataSet.add(old_sum, current)
(sum map (new RichMapFunction[Vector, Vector]{
def map(x: Vector) = new
Vector(getIterationRuntimeContext.getSuperstepNumber, x.values)
}),
sum)
}
Error:
10/14/2014 15:57:35: Job execution switched to status RUNNING
10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to SCHEDULED
10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to DEPLOYING
10/14/2014 15:57:35: DataSource ([]) (1/1) switched to SCHEDULED
10/14/2014 15:57:35: DataSource ([]) (1/1) switched to DEPLOYING
10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat (/tmp/tmpBhOsLd) -
UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched
to SCHEDULED
10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat (/tmp/tmpBhOsLd) -
UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched
to DEPLOYING
10/14/2014 15:57:35: DataSource ([-1 0.0]) (1/1) switched to RUNNING
10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
Iteration)) (1/1) switched to SCHEDULED
10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
Iteration)) (1/1) switched to DEPLOYING
10/14/2014 15:57:35: CHAIN DataSource (TextInputFormat (/tmp/tmpBhOsLd) -
UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched
to RUNNING
10/14/2014 15:57:35: DataSource ([]) (1/1) switched to RUNNING
10/14/2014 15:57:35: IterationHead(WorksetIteration (Unnamed Delta
Iteration)) (1/1) switched to RUNNING
10/14/2014 15:57:35: CHAIN
Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
(org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
(bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to SCHEDULED
10/14/2014 15:57:35: CHAIN
Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
(org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
(bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to DEPLOYING
10/14/2014 15:57:36: CHAIN
Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
(org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to SCHEDULED
10/14/2014 15:57:36: CHAIN
Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
(org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to DEPLOYING
10/14/2014 15:57:36: CHAIN
Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
(org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
(bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to RUNNING
10/14/2014 15:57:36: CHAIN
Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
(org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to RUNNING
10/14/2014 15:57:36: CHAIN
Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
(org.apache.flink.api.scala.DataSet$$anon$1) -> Filter
(bumpboost.BumpBoost$$anonfun$8$$anon$1) (1/1) switched to FAILED
java.lang.IllegalStateException: This stub is not part of an iteration step
function.
at
org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
at
org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
at
org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
at
org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
at
org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
at
org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
at org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
at
org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
at java.lang.Thread.run(Thread.java:745)
10/14/2014 15:57:36: Job execution switched to status FAILING
10/14/2014 15:57:36: DataSource ([]) (1/1) switched to CANCELING
10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat (/tmp/tmpBhOsLd) -
UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched
to CANCELING
10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to CANCELING
10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed Delta
Iteration)) (1/1) switched to CANCELING
10/14/2014 15:57:36: Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1)
switched to CANCELED
10/14/2014 15:57:36: DataSink(TextOutputFormat (/tmp/tmplSYJ7S) - UTF-8)
(1/1) switched to CANCELED
10/14/2014 15:57:36: Sync (WorksetIteration (Unnamed Delta Iteration))
(1/1) switched to CANCELED
10/14/2014 15:57:36: CHAIN
Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
(org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to CANCELING
10/14/2014 15:57:36: Map (bumpboost.BumpBoost$$anonfun$8$$anon$2) (1/1)
switched to CANCELED
10/14/2014 15:57:36: SolutionSet Delta (1/1) switched to CANCELED
10/14/2014 15:57:36: DataSource ([]) (1/1) switched to CANCELED
10/14/2014 15:57:36: CHAIN DataSource (TextInputFormat (/tmp/tmpBhOsLd) -
UTF-8) -> Map (org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched
to CANCELED
10/14/2014 15:57:36: DataSource ([-1 0.0]) (1/1) switched to CANCELED
10/14/2014 15:57:36: CHAIN
Join(org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5) -> Map
(org.apache.flink.api.scala.DataSet$$anon$1) (1/1) switched to CANCELED
10/14/2014 15:57:36: IterationHead(WorksetIteration (Unnamed Delta
Iteration)) (1/1) switched to CANCELED
10/14/2014 15:57:36: Job execution switched to status FAILED
Error: The program execution failed: java.lang.IllegalStateException: This
stub is not part of an iteration step function.
at
org.apache.flink.api.common.functions.AbstractRichFunction.getIterationRuntimeContext(AbstractRichFunction.java:59)
at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:40)
at bumpboost.BumpBoost$$anonfun$8$$anon$1.filter(BumpBoost.scala:39)
at
org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:47)
at
org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
at
org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
at
org.apache.flink.api.scala.UnfinishedJoinOperation$$anon$5.join(joinDataSet.scala:184)
at
org.apache.flink.runtime.operators.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:140)
at org.apache.flink.runtime.operators.MatchDriver.run(MatchDriver.java:148)
at
org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:484)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:359)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:235)
at java.lang.Thread.run(Thread.java:745)
On Tue, Oct 14, 2014 at 3:32 PM, Maximilian Alber <
alber.maximilian@gmail.com> wrote:
> Should work now.
> Cheers
>
> On Fri, Oct 10, 2014 at 3:38 PM, Maximilian Alber <
> alber.maximilian@gmail.com> wrote:
>
>> Ok, thanks.
>> Please let me know when it is fixed.
>>
>> Cheers
>> Max
>>
>> On Fri, Oct 10, 2014 at 1:34 PM, Stephan Ewen <se...@apache.org> wrote:
>>
>>> Thank you, I will look into that...
>>>
>>
>>
>
Re: Forced to use Solution Set in Step Function
Posted by Maximilian Alber <al...@gmail.com>.
Should work now.
Cheers
On Fri, Oct 10, 2014 at 3:38 PM, Maximilian Alber <
alber.maximilian@gmail.com> wrote:
> Ok, thanks.
> Please let me know when it is fixed.
>
> Cheers
> Max
>
> On Fri, Oct 10, 2014 at 1:34 PM, Stephan Ewen <se...@apache.org> wrote:
>
>> Thank you, I will look into that...
>>
>
>
Re: Forced to use Solution Set in Step Function
Posted by Maximilian Alber <al...@gmail.com>.
Ok, thanks.
Please let me know when it is fixed.
Cheers
Max
On Fri, Oct 10, 2014 at 1:34 PM, Stephan Ewen <se...@apache.org> wrote:
> Thank you, I will look into that...
>
Re: Forced to use Solution Set in Step Function
Posted by Stephan Ewen <se...@apache.org>.
Thank you, I will look into that...
Re: Forced to use Solution Set in Step Function
Posted by Maximilian Alber <al...@gmail.com>.
I opend an issue.
After compiling with the commented check (actually the whole block, line
867-900) I get a NullPointerException:
java.lang.NullPointerException
at
org.apache.flink.compiler.PactCompiler$GraphCreatingVisitor.postVisit(PactCompiler.java:870)
at
org.apache.flink.compiler.PactCompiler$GraphCreatingVisitor.postVisit(PactCompiler.java:622)
at
org.apache.flink.api.common.operators.DualInputOperator.accept(DualInputOperator.java:283)
at
org.apache.flink.api.common.operators.SingleInputOperator.accept(SingleInputOperator.java:202)
at
org.apache.flink.api.common.operators.GenericDataSinkBase.accept(GenericDataSinkBase.java:286)
at org.apache.flink.api.common.Plan.accept(Plan.java:281)
at org.apache.flink.compiler.PactCompiler.compile(PactCompiler.java:517)
at org.apache.flink.compiler.PactCompiler.compile(PactCompiler.java:466)
at org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:196)
at org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:209)
at org.apache.flink.client.program.Client.run(Client.java:285)
at org.apache.flink.client.program.Client.run(Client.java:230)
at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:347)
at org.apache.flink.client.CliFrontend.run(CliFrontend.java:334)
at
org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1001)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1025)
On Wed, Oct 8, 2014 at 5:11 PM, Maximilian Alber <alber.maximilian@gmail.com
> wrote:
> I will tomorrow.
> I will try. Thanks.
>
> Cheers
> Max
>
> On Wed, Oct 8, 2014 at 2:52 PM, Stephan Ewen <se...@apache.org> wrote:
>
>> I think we can disable the check. Can you open an issue for that?
>>
>> As a quick fix, is it possible for you to compile yourself a version
>> where the check is commented out?
>>
>
>
Re: Forced to use Solution Set in Step Function
Posted by Maximilian Alber <al...@gmail.com>.
I will tomorrow.
I will try. Thanks.
Cheers
Max
On Wed, Oct 8, 2014 at 2:52 PM, Stephan Ewen <se...@apache.org> wrote:
> I think we can disable the check. Can you open an issue for that?
>
> As a quick fix, is it possible for you to compile yourself a version where
> the check is commented out?
>
Re: Forced to use Solution Set in Step Function
Posted by Stephan Ewen <se...@apache.org>.
I think we can disable the check. Can you open an issue for that?
As a quick fix, is it possible for you to compile yourself a version where
the check is commented out?
Re: Forced to use Solution Set in Step Function
Posted by Aljoscha Krettek <al...@apache.org>.
Correct, no changing. DataSets can never be modified, in fact. (Except
the solution set in a delta iteration, but there the user is also not
explicitly changing a DataSet.)
On Wed, Oct 8, 2014 at 12:56 PM, Maximilian Alber
<al...@gmail.com> wrote:
> But no changing?
>
> On Wed, Oct 8, 2014 at 11:39 AM, Aljoscha Krettek <al...@apache.org>
> wrote:
>>
>> Yes, you can refer to outside datasets in an iteration.
>>
>> On Wed, Oct 8, 2014 at 10:37 AM, Maximilian Alber
>> <al...@gmail.com> wrote:
>> > Just got into my mind: it possible to have broadcast sets inside the
>> > iteration functions with datasets which are "located" outside of it (via
>> > closure)?
>> >
>> > The basic type of my iteration is that I have a datasets which gets
>> > altered
>> > and is needed each iterations aka working set, in my case I have also a
>> > constant dataset which gets not modified (that messes up the code) and a
>> > resulting dataset which is not needed inside the step function.
>> > Thus similar to iterate with delta.
>> >
>> > Cheers,
>> > Max
>> >
>> >
>> > On Wed, Oct 8, 2014 at 10:26 AM, Aljoscha Krettek <al...@apache.org>
>> > wrote:
>> >>
>> >> Hmm, what it really needs is a different kind of iteration primitive.
>> >> Basically a bulk iteration where you can output values in each
>> >> iteration that get collected.
>> >>
>> >> On Wed, Oct 8, 2014 at 10:02 AM, Maximilian Alber
>> >> <al...@gmail.com> wrote:
>> >> > Hi!
>> >> >
>> >> > Hmm I don't think so. I have two datasets, which I cannot really
>> >> > merge
>> >> > together. After some thinking this solution was the only I got for
>> >> > solving
>> >> > my problem:
>> >> > I have a DataSet with Vector(in this case just with length one) each
>> >> > has
>> >> > an
>> >> > id and an array with values. Out of that I would like to create the
>> >> > prefix
>> >> > sums aka the cumulative sums. To do it I need the to keep the dataset
>> >> > with
>> >> > the vectors and the dataset where I store the sums.
>> >> >
>> >> > In the Scala version I could use a dataset inside the iteration
>> >> > without
>> >> > passing as solution or workset just via closures?
>> >> >
>> >> > Maybe a flag to disable the check would be suitable?
>> >> >
>> >> > Thanks!
>> >> > Cheers,
>> >> > Max
>> >> >
>> >> > On Tue, Oct 7, 2014 at 4:34 PM, Stephan Ewen <se...@apache.org>
>> >> > wrote:
>> >> >>
>> >> >> Hey!
>> >> >>
>> >> >> Is the algorithm you are using a delta iteration in fact. If you
>> >> >> actually
>> >> >> do not use the solution set, can you model it as a bulk-iteration?
>> >> >>
>> >> >> If you actually need the solution set to accumulate data, we can
>> >> >> probably
>> >> >> deactivate that check in the compiler. As far as I remember, there
>> >> >> is
>> >> >> no
>> >> >> requirement in the runtime to join with the solution set. The check
>> >> >> is
>> >> >> meant
>> >> >> to help programmers that forgot the join...
>> >> >>
>> >> >> Greetings,
>> >> >> Stephan
>> >> >>
>> >> >>
>> >> >>
>> >> >> On Tue, Oct 7, 2014 at 3:13 PM, Maximilian Alber
>> >> >> <al...@gmail.com> wrote:
>> >> >>>
>> >> >>> Hi Flinksters!
>> >> >>>
>> >> >>> I would like to use iterateDelta function. I don't need the
>> >> >>> solution
>> >> >>> set
>> >> >>> inside the step function, because I generate a different values out
>> >> >>> of
>> >> >>> the
>> >> >>> working set. Unfortunately the compiler of the development version
>> >> >>> doesn't
>> >> >>> like that. Is there a workaround?
>> >> >>>
>> >> >>> The code:
>> >> >>>
>> >> >>> val residual_2a = residual_2 union
>> >> >>> env.fromCollection(Seq(Vector.zeros(config.dimensions)))
>> >> >>> val emptyDataSet = env.fromCollection[Vector](Seq())
>> >> >>> val cumSum = emptyDataSet.iterateDelta(residual_2a, 1000000,
>> >> >>> Array("id"))
>> >> >>> {
>> >> >>> (solutionset, workset) =>
>> >> >>> val old_sum = workset filter {_.id == -1}
>> >> >>> val current = workset filter (new RichFilterFunction[Vector]{
>> >> >>> def filter(x: Vector) = x.id ==
>> >> >>> (getIterationRuntimeContext.getSuperstepNumber)
>> >> >>> })
>> >> >>> val residual_2 = workset filter {_.id != -1}
>> >> >>> val sum = VectorDataSet.add(old_sum, current)
>> >> >>>
>> >> >>> (sum map (new RichMapFunction[Vector, Vector]{
>> >> >>> def map(x: Vector) = new
>> >> >>> Vector(getIterationRuntimeContext.getSuperstepNumber, x.values)
>> >> >>> }),
>> >> >>> residual_2 union sum)
>> >> >>> }
>> >> >>>
>> >> >>> The error:
>> >> >>>
>> >> >>> org.apache.flink.compiler.CompilerException: Error: The step
>> >> >>> function
>> >> >>> does not reference the solution set.
>> >> >>> at
>> >> >>>
>> >> >>>
>> >> >>> org.apache.flink.compiler.PactCompiler$GraphCreatingVisitor.postVisit(PactCompiler.java:868)
>> >> >>> at
>> >> >>>
>> >> >>>
>> >> >>> org.apache.flink.compiler.PactCompiler$GraphCreatingVisitor.postVisit(PactCompiler.java:622)
>> >> >>> at
>> >> >>>
>> >> >>>
>> >> >>> org.apache.flink.api.common.operators.DualInputOperator.accept(DualInputOperator.java:283)
>> >> >>> at
>> >> >>>
>> >> >>>
>> >> >>> org.apache.flink.api.common.operators.SingleInputOperator.accept(SingleInputOperator.java:202)
>> >> >>> at
>> >> >>>
>> >> >>>
>> >> >>> org.apache.flink.api.common.operators.GenericDataSinkBase.accept(GenericDataSinkBase.java:286)
>> >> >>> at org.apache.flink.api.common.Plan.accept(Plan.java:281)
>> >> >>> at
>> >> >>>
>> >> >>> org.apache.flink.compiler.PactCompiler.compile(PactCompiler.java:517)
>> >> >>> at
>> >> >>>
>> >> >>> org.apache.flink.compiler.PactCompiler.compile(PactCompiler.java:466)
>> >> >>> at
>> >> >>>
>> >> >>>
>> >> >>> org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:196)
>> >> >>> at
>> >> >>>
>> >> >>>
>> >> >>> org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:209)
>> >> >>> at org.apache.flink.client.program.Client.run(Client.java:285)
>> >> >>> at org.apache.flink.client.program.Client.run(Client.java:230)
>> >> >>> at
>> >> >>>
>> >> >>>
>> >> >>> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:347)
>> >> >>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:334)
>> >> >>> at
>> >> >>>
>> >> >>>
>> >> >>> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1001)
>> >> >>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1025)
>> >> >>>
>> >> >>> Thanks!
>> >> >>> Cheers,
>> >> >>> Max
>> >> >>
>> >> >>
>> >> >
>> >
>> >
>
>
Re: Forced to use Solution Set in Step Function
Posted by Maximilian Alber <al...@gmail.com>.
But no changing?
On Wed, Oct 8, 2014 at 11:39 AM, Aljoscha Krettek <al...@apache.org>
wrote:
> Yes, you can refer to outside datasets in an iteration.
>
> On Wed, Oct 8, 2014 at 10:37 AM, Maximilian Alber
> <al...@gmail.com> wrote:
> > Just got into my mind: it possible to have broadcast sets inside the
> > iteration functions with datasets which are "located" outside of it (via
> > closure)?
> >
> > The basic type of my iteration is that I have a datasets which gets
> altered
> > and is needed each iterations aka working set, in my case I have also a
> > constant dataset which gets not modified (that messes up the code) and a
> > resulting dataset which is not needed inside the step function.
> > Thus similar to iterate with delta.
> >
> > Cheers,
> > Max
> >
> >
> > On Wed, Oct 8, 2014 at 10:26 AM, Aljoscha Krettek <al...@apache.org>
> > wrote:
> >>
> >> Hmm, what it really needs is a different kind of iteration primitive.
> >> Basically a bulk iteration where you can output values in each
> >> iteration that get collected.
> >>
> >> On Wed, Oct 8, 2014 at 10:02 AM, Maximilian Alber
> >> <al...@gmail.com> wrote:
> >> > Hi!
> >> >
> >> > Hmm I don't think so. I have two datasets, which I cannot really merge
> >> > together. After some thinking this solution was the only I got for
> >> > solving
> >> > my problem:
> >> > I have a DataSet with Vector(in this case just with length one) each
> has
> >> > an
> >> > id and an array with values. Out of that I would like to create the
> >> > prefix
> >> > sums aka the cumulative sums. To do it I need the to keep the dataset
> >> > with
> >> > the vectors and the dataset where I store the sums.
> >> >
> >> > In the Scala version I could use a dataset inside the iteration
> without
> >> > passing as solution or workset just via closures?
> >> >
> >> > Maybe a flag to disable the check would be suitable?
> >> >
> >> > Thanks!
> >> > Cheers,
> >> > Max
> >> >
> >> > On Tue, Oct 7, 2014 at 4:34 PM, Stephan Ewen <se...@apache.org>
> wrote:
> >> >>
> >> >> Hey!
> >> >>
> >> >> Is the algorithm you are using a delta iteration in fact. If you
> >> >> actually
> >> >> do not use the solution set, can you model it as a bulk-iteration?
> >> >>
> >> >> If you actually need the solution set to accumulate data, we can
> >> >> probably
> >> >> deactivate that check in the compiler. As far as I remember, there is
> >> >> no
> >> >> requirement in the runtime to join with the solution set. The check
> is
> >> >> meant
> >> >> to help programmers that forgot the join...
> >> >>
> >> >> Greetings,
> >> >> Stephan
> >> >>
> >> >>
> >> >>
> >> >> On Tue, Oct 7, 2014 at 3:13 PM, Maximilian Alber
> >> >> <al...@gmail.com> wrote:
> >> >>>
> >> >>> Hi Flinksters!
> >> >>>
> >> >>> I would like to use iterateDelta function. I don't need the solution
> >> >>> set
> >> >>> inside the step function, because I generate a different values out
> of
> >> >>> the
> >> >>> working set. Unfortunately the compiler of the development version
> >> >>> doesn't
> >> >>> like that. Is there a workaround?
> >> >>>
> >> >>> The code:
> >> >>>
> >> >>> val residual_2a = residual_2 union
> >> >>> env.fromCollection(Seq(Vector.zeros(config.dimensions)))
> >> >>> val emptyDataSet = env.fromCollection[Vector](Seq())
> >> >>> val cumSum = emptyDataSet.iterateDelta(residual_2a, 1000000,
> >> >>> Array("id"))
> >> >>> {
> >> >>> (solutionset, workset) =>
> >> >>> val old_sum = workset filter {_.id == -1}
> >> >>> val current = workset filter (new RichFilterFunction[Vector]{
> >> >>> def filter(x: Vector) = x.id ==
> >> >>> (getIterationRuntimeContext.getSuperstepNumber)
> >> >>> })
> >> >>> val residual_2 = workset filter {_.id != -1}
> >> >>> val sum = VectorDataSet.add(old_sum, current)
> >> >>>
> >> >>> (sum map (new RichMapFunction[Vector, Vector]{
> >> >>> def map(x: Vector) = new
> >> >>> Vector(getIterationRuntimeContext.getSuperstepNumber, x.values)
> >> >>> }),
> >> >>> residual_2 union sum)
> >> >>> }
> >> >>>
> >> >>> The error:
> >> >>>
> >> >>> org.apache.flink.compiler.CompilerException: Error: The step
> function
> >> >>> does not reference the solution set.
> >> >>> at
> >> >>>
> >> >>>
> org.apache.flink.compiler.PactCompiler$GraphCreatingVisitor.postVisit(PactCompiler.java:868)
> >> >>> at
> >> >>>
> >> >>>
> org.apache.flink.compiler.PactCompiler$GraphCreatingVisitor.postVisit(PactCompiler.java:622)
> >> >>> at
> >> >>>
> >> >>>
> org.apache.flink.api.common.operators.DualInputOperator.accept(DualInputOperator.java:283)
> >> >>> at
> >> >>>
> >> >>>
> org.apache.flink.api.common.operators.SingleInputOperator.accept(SingleInputOperator.java:202)
> >> >>> at
> >> >>>
> >> >>>
> org.apache.flink.api.common.operators.GenericDataSinkBase.accept(GenericDataSinkBase.java:286)
> >> >>> at org.apache.flink.api.common.Plan.accept(Plan.java:281)
> >> >>> at
> >> >>>
> org.apache.flink.compiler.PactCompiler.compile(PactCompiler.java:517)
> >> >>> at
> >> >>>
> org.apache.flink.compiler.PactCompiler.compile(PactCompiler.java:466)
> >> >>> at
> >> >>>
> >> >>>
> org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:196)
> >> >>> at
> >> >>>
> >> >>>
> org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:209)
> >> >>> at org.apache.flink.client.program.Client.run(Client.java:285)
> >> >>> at org.apache.flink.client.program.Client.run(Client.java:230)
> >> >>> at
> >> >>>
> >> >>>
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:347)
> >> >>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:334)
> >> >>> at
> >> >>>
> >> >>>
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1001)
> >> >>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1025)
> >> >>>
> >> >>> Thanks!
> >> >>> Cheers,
> >> >>> Max
> >> >>
> >> >>
> >> >
> >
> >
>
Re: Forced to use Solution Set in Step Function
Posted by Aljoscha Krettek <al...@apache.org>.
Yes, you can refer to outside datasets in an iteration.
On Wed, Oct 8, 2014 at 10:37 AM, Maximilian Alber
<al...@gmail.com> wrote:
> Just got into my mind: it possible to have broadcast sets inside the
> iteration functions with datasets which are "located" outside of it (via
> closure)?
>
> The basic type of my iteration is that I have a datasets which gets altered
> and is needed each iterations aka working set, in my case I have also a
> constant dataset which gets not modified (that messes up the code) and a
> resulting dataset which is not needed inside the step function.
> Thus similar to iterate with delta.
>
> Cheers,
> Max
>
>
> On Wed, Oct 8, 2014 at 10:26 AM, Aljoscha Krettek <al...@apache.org>
> wrote:
>>
>> Hmm, what it really needs is a different kind of iteration primitive.
>> Basically a bulk iteration where you can output values in each
>> iteration that get collected.
>>
>> On Wed, Oct 8, 2014 at 10:02 AM, Maximilian Alber
>> <al...@gmail.com> wrote:
>> > Hi!
>> >
>> > Hmm I don't think so. I have two datasets, which I cannot really merge
>> > together. After some thinking this solution was the only I got for
>> > solving
>> > my problem:
>> > I have a DataSet with Vector(in this case just with length one) each has
>> > an
>> > id and an array with values. Out of that I would like to create the
>> > prefix
>> > sums aka the cumulative sums. To do it I need the to keep the dataset
>> > with
>> > the vectors and the dataset where I store the sums.
>> >
>> > In the Scala version I could use a dataset inside the iteration without
>> > passing as solution or workset just via closures?
>> >
>> > Maybe a flag to disable the check would be suitable?
>> >
>> > Thanks!
>> > Cheers,
>> > Max
>> >
>> > On Tue, Oct 7, 2014 at 4:34 PM, Stephan Ewen <se...@apache.org> wrote:
>> >>
>> >> Hey!
>> >>
>> >> Is the algorithm you are using a delta iteration in fact. If you
>> >> actually
>> >> do not use the solution set, can you model it as a bulk-iteration?
>> >>
>> >> If you actually need the solution set to accumulate data, we can
>> >> probably
>> >> deactivate that check in the compiler. As far as I remember, there is
>> >> no
>> >> requirement in the runtime to join with the solution set. The check is
>> >> meant
>> >> to help programmers that forgot the join...
>> >>
>> >> Greetings,
>> >> Stephan
>> >>
>> >>
>> >>
>> >> On Tue, Oct 7, 2014 at 3:13 PM, Maximilian Alber
>> >> <al...@gmail.com> wrote:
>> >>>
>> >>> Hi Flinksters!
>> >>>
>> >>> I would like to use iterateDelta function. I don't need the solution
>> >>> set
>> >>> inside the step function, because I generate a different values out of
>> >>> the
>> >>> working set. Unfortunately the compiler of the development version
>> >>> doesn't
>> >>> like that. Is there a workaround?
>> >>>
>> >>> The code:
>> >>>
>> >>> val residual_2a = residual_2 union
>> >>> env.fromCollection(Seq(Vector.zeros(config.dimensions)))
>> >>> val emptyDataSet = env.fromCollection[Vector](Seq())
>> >>> val cumSum = emptyDataSet.iterateDelta(residual_2a, 1000000,
>> >>> Array("id"))
>> >>> {
>> >>> (solutionset, workset) =>
>> >>> val old_sum = workset filter {_.id == -1}
>> >>> val current = workset filter (new RichFilterFunction[Vector]{
>> >>> def filter(x: Vector) = x.id ==
>> >>> (getIterationRuntimeContext.getSuperstepNumber)
>> >>> })
>> >>> val residual_2 = workset filter {_.id != -1}
>> >>> val sum = VectorDataSet.add(old_sum, current)
>> >>>
>> >>> (sum map (new RichMapFunction[Vector, Vector]{
>> >>> def map(x: Vector) = new
>> >>> Vector(getIterationRuntimeContext.getSuperstepNumber, x.values)
>> >>> }),
>> >>> residual_2 union sum)
>> >>> }
>> >>>
>> >>> The error:
>> >>>
>> >>> org.apache.flink.compiler.CompilerException: Error: The step function
>> >>> does not reference the solution set.
>> >>> at
>> >>>
>> >>> org.apache.flink.compiler.PactCompiler$GraphCreatingVisitor.postVisit(PactCompiler.java:868)
>> >>> at
>> >>>
>> >>> org.apache.flink.compiler.PactCompiler$GraphCreatingVisitor.postVisit(PactCompiler.java:622)
>> >>> at
>> >>>
>> >>> org.apache.flink.api.common.operators.DualInputOperator.accept(DualInputOperator.java:283)
>> >>> at
>> >>>
>> >>> org.apache.flink.api.common.operators.SingleInputOperator.accept(SingleInputOperator.java:202)
>> >>> at
>> >>>
>> >>> org.apache.flink.api.common.operators.GenericDataSinkBase.accept(GenericDataSinkBase.java:286)
>> >>> at org.apache.flink.api.common.Plan.accept(Plan.java:281)
>> >>> at
>> >>> org.apache.flink.compiler.PactCompiler.compile(PactCompiler.java:517)
>> >>> at
>> >>> org.apache.flink.compiler.PactCompiler.compile(PactCompiler.java:466)
>> >>> at
>> >>>
>> >>> org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:196)
>> >>> at
>> >>>
>> >>> org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:209)
>> >>> at org.apache.flink.client.program.Client.run(Client.java:285)
>> >>> at org.apache.flink.client.program.Client.run(Client.java:230)
>> >>> at
>> >>>
>> >>> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:347)
>> >>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:334)
>> >>> at
>> >>>
>> >>> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1001)
>> >>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1025)
>> >>>
>> >>> Thanks!
>> >>> Cheers,
>> >>> Max
>> >>
>> >>
>> >
>
>
Re: Forced to use Solution Set in Step Function
Posted by Maximilian Alber <al...@gmail.com>.
Just got into my mind: it possible to have broadcast sets inside the
iteration functions with datasets which are "located" outside of it (via
closure)?
The basic type of my iteration is that I have a datasets which gets altered
and is needed each iterations aka working set, in my case I have also a
constant dataset which gets not modified (that messes up the code) and a
resulting dataset which is not needed inside the step function.
Thus similar to iterate with delta.
Cheers,
Max
On Wed, Oct 8, 2014 at 10:26 AM, Aljoscha Krettek <al...@apache.org>
wrote:
> Hmm, what it really needs is a different kind of iteration primitive.
> Basically a bulk iteration where you can output values in each
> iteration that get collected.
>
> On Wed, Oct 8, 2014 at 10:02 AM, Maximilian Alber
> <al...@gmail.com> wrote:
> > Hi!
> >
> > Hmm I don't think so. I have two datasets, which I cannot really merge
> > together. After some thinking this solution was the only I got for
> solving
> > my problem:
> > I have a DataSet with Vector(in this case just with length one) each has
> an
> > id and an array with values. Out of that I would like to create the
> prefix
> > sums aka the cumulative sums. To do it I need the to keep the dataset
> with
> > the vectors and the dataset where I store the sums.
> >
> > In the Scala version I could use a dataset inside the iteration without
> > passing as solution or workset just via closures?
> >
> > Maybe a flag to disable the check would be suitable?
> >
> > Thanks!
> > Cheers,
> > Max
> >
> > On Tue, Oct 7, 2014 at 4:34 PM, Stephan Ewen <se...@apache.org> wrote:
> >>
> >> Hey!
> >>
> >> Is the algorithm you are using a delta iteration in fact. If you
> actually
> >> do not use the solution set, can you model it as a bulk-iteration?
> >>
> >> If you actually need the solution set to accumulate data, we can
> probably
> >> deactivate that check in the compiler. As far as I remember, there is no
> >> requirement in the runtime to join with the solution set. The check is
> meant
> >> to help programmers that forgot the join...
> >>
> >> Greetings,
> >> Stephan
> >>
> >>
> >>
> >> On Tue, Oct 7, 2014 at 3:13 PM, Maximilian Alber
> >> <al...@gmail.com> wrote:
> >>>
> >>> Hi Flinksters!
> >>>
> >>> I would like to use iterateDelta function. I don't need the solution
> set
> >>> inside the step function, because I generate a different values out of
> the
> >>> working set. Unfortunately the compiler of the development version
> doesn't
> >>> like that. Is there a workaround?
> >>>
> >>> The code:
> >>>
> >>> val residual_2a = residual_2 union
> >>> env.fromCollection(Seq(Vector.zeros(config.dimensions)))
> >>> val emptyDataSet = env.fromCollection[Vector](Seq())
> >>> val cumSum = emptyDataSet.iterateDelta(residual_2a, 1000000,
> Array("id"))
> >>> {
> >>> (solutionset, workset) =>
> >>> val old_sum = workset filter {_.id == -1}
> >>> val current = workset filter (new RichFilterFunction[Vector]{
> >>> def filter(x: Vector) = x.id ==
> >>> (getIterationRuntimeContext.getSuperstepNumber)
> >>> })
> >>> val residual_2 = workset filter {_.id != -1}
> >>> val sum = VectorDataSet.add(old_sum, current)
> >>>
> >>> (sum map (new RichMapFunction[Vector, Vector]{
> >>> def map(x: Vector) = new
> >>> Vector(getIterationRuntimeContext.getSuperstepNumber, x.values)
> >>> }),
> >>> residual_2 union sum)
> >>> }
> >>>
> >>> The error:
> >>>
> >>> org.apache.flink.compiler.CompilerException: Error: The step function
> >>> does not reference the solution set.
> >>> at
> >>>
> org.apache.flink.compiler.PactCompiler$GraphCreatingVisitor.postVisit(PactCompiler.java:868)
> >>> at
> >>>
> org.apache.flink.compiler.PactCompiler$GraphCreatingVisitor.postVisit(PactCompiler.java:622)
> >>> at
> >>>
> org.apache.flink.api.common.operators.DualInputOperator.accept(DualInputOperator.java:283)
> >>> at
> >>>
> org.apache.flink.api.common.operators.SingleInputOperator.accept(SingleInputOperator.java:202)
> >>> at
> >>>
> org.apache.flink.api.common.operators.GenericDataSinkBase.accept(GenericDataSinkBase.java:286)
> >>> at org.apache.flink.api.common.Plan.accept(Plan.java:281)
> >>> at
> org.apache.flink.compiler.PactCompiler.compile(PactCompiler.java:517)
> >>> at
> org.apache.flink.compiler.PactCompiler.compile(PactCompiler.java:466)
> >>> at
> >>>
> org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:196)
> >>> at
> >>>
> org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:209)
> >>> at org.apache.flink.client.program.Client.run(Client.java:285)
> >>> at org.apache.flink.client.program.Client.run(Client.java:230)
> >>> at
> >>>
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:347)
> >>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:334)
> >>> at
> >>>
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1001)
> >>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1025)
> >>>
> >>> Thanks!
> >>> Cheers,
> >>> Max
> >>
> >>
> >
>
Re: Forced to use Solution Set in Step Function
Posted by Aljoscha Krettek <al...@apache.org>.
Hmm, what it really needs is a different kind of iteration primitive.
Basically a bulk iteration where you can output values in each
iteration that get collected.
On Wed, Oct 8, 2014 at 10:02 AM, Maximilian Alber
<al...@gmail.com> wrote:
> Hi!
>
> Hmm I don't think so. I have two datasets, which I cannot really merge
> together. After some thinking this solution was the only I got for solving
> my problem:
> I have a DataSet with Vector(in this case just with length one) each has an
> id and an array with values. Out of that I would like to create the prefix
> sums aka the cumulative sums. To do it I need the to keep the dataset with
> the vectors and the dataset where I store the sums.
>
> In the Scala version I could use a dataset inside the iteration without
> passing as solution or workset just via closures?
>
> Maybe a flag to disable the check would be suitable?
>
> Thanks!
> Cheers,
> Max
>
> On Tue, Oct 7, 2014 at 4:34 PM, Stephan Ewen <se...@apache.org> wrote:
>>
>> Hey!
>>
>> Is the algorithm you are using a delta iteration in fact. If you actually
>> do not use the solution set, can you model it as a bulk-iteration?
>>
>> If you actually need the solution set to accumulate data, we can probably
>> deactivate that check in the compiler. As far as I remember, there is no
>> requirement in the runtime to join with the solution set. The check is meant
>> to help programmers that forgot the join...
>>
>> Greetings,
>> Stephan
>>
>>
>>
>> On Tue, Oct 7, 2014 at 3:13 PM, Maximilian Alber
>> <al...@gmail.com> wrote:
>>>
>>> Hi Flinksters!
>>>
>>> I would like to use iterateDelta function. I don't need the solution set
>>> inside the step function, because I generate a different values out of the
>>> working set. Unfortunately the compiler of the development version doesn't
>>> like that. Is there a workaround?
>>>
>>> The code:
>>>
>>> val residual_2a = residual_2 union
>>> env.fromCollection(Seq(Vector.zeros(config.dimensions)))
>>> val emptyDataSet = env.fromCollection[Vector](Seq())
>>> val cumSum = emptyDataSet.iterateDelta(residual_2a, 1000000, Array("id"))
>>> {
>>> (solutionset, workset) =>
>>> val old_sum = workset filter {_.id == -1}
>>> val current = workset filter (new RichFilterFunction[Vector]{
>>> def filter(x: Vector) = x.id ==
>>> (getIterationRuntimeContext.getSuperstepNumber)
>>> })
>>> val residual_2 = workset filter {_.id != -1}
>>> val sum = VectorDataSet.add(old_sum, current)
>>>
>>> (sum map (new RichMapFunction[Vector, Vector]{
>>> def map(x: Vector) = new
>>> Vector(getIterationRuntimeContext.getSuperstepNumber, x.values)
>>> }),
>>> residual_2 union sum)
>>> }
>>>
>>> The error:
>>>
>>> org.apache.flink.compiler.CompilerException: Error: The step function
>>> does not reference the solution set.
>>> at
>>> org.apache.flink.compiler.PactCompiler$GraphCreatingVisitor.postVisit(PactCompiler.java:868)
>>> at
>>> org.apache.flink.compiler.PactCompiler$GraphCreatingVisitor.postVisit(PactCompiler.java:622)
>>> at
>>> org.apache.flink.api.common.operators.DualInputOperator.accept(DualInputOperator.java:283)
>>> at
>>> org.apache.flink.api.common.operators.SingleInputOperator.accept(SingleInputOperator.java:202)
>>> at
>>> org.apache.flink.api.common.operators.GenericDataSinkBase.accept(GenericDataSinkBase.java:286)
>>> at org.apache.flink.api.common.Plan.accept(Plan.java:281)
>>> at org.apache.flink.compiler.PactCompiler.compile(PactCompiler.java:517)
>>> at org.apache.flink.compiler.PactCompiler.compile(PactCompiler.java:466)
>>> at
>>> org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:196)
>>> at
>>> org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:209)
>>> at org.apache.flink.client.program.Client.run(Client.java:285)
>>> at org.apache.flink.client.program.Client.run(Client.java:230)
>>> at
>>> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:347)
>>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:334)
>>> at
>>> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1001)
>>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1025)
>>>
>>> Thanks!
>>> Cheers,
>>> Max
>>
>>
>
Re: Forced to use Solution Set in Step Function
Posted by Maximilian Alber <al...@gmail.com>.
Hi!
Hmm I don't think so. I have two datasets, which I cannot really merge
together. After some thinking this solution was the only I got for solving
my problem:
I have a DataSet with Vector(in this case just with length one) each has an
id and an array with values. Out of that I would like to create the prefix
sums aka the cumulative sums. To do it I need the to keep the dataset with
the vectors and the dataset where I store the sums.
In the Scala version I could use a dataset inside the iteration without
passing as solution or workset just via closures?
Maybe a flag to disable the check would be suitable?
Thanks!
Cheers,
Max
On Tue, Oct 7, 2014 at 4:34 PM, Stephan Ewen <se...@apache.org> wrote:
> Hey!
>
> Is the algorithm you are using a delta iteration in fact. If you actually
> do not use the solution set, can you model it as a bulk-iteration?
>
> If you actually need the solution set to accumulate data, we can probably
> deactivate that check in the compiler. As far as I remember, there is no
> requirement in the runtime to join with the solution set. The check is
> meant to help programmers that forgot the join...
>
> Greetings,
> Stephan
>
>
>
> On Tue, Oct 7, 2014 at 3:13 PM, Maximilian Alber <
> alber.maximilian@gmail.com> wrote:
>
>> Hi Flinksters!
>>
>> I would like to use iterateDelta function. I don't need the solution set
>> inside the step function, because I generate a different values out of the
>> working set. Unfortunately the compiler of the development version doesn't
>> like that. Is there a workaround?
>>
>> The code:
>>
>> val residual_2a = residual_2 union
>> env.fromCollection(Seq(Vector.zeros(config.dimensions)))
>> val emptyDataSet = env.fromCollection[Vector](Seq())
>> val cumSum = emptyDataSet.iterateDelta(residual_2a, 1000000, Array("id"))
>> {
>> (solutionset, workset) =>
>> val old_sum = workset filter {_.id == -1}
>> val current = workset filter (new RichFilterFunction[Vector]{
>> def filter(x: Vector) = x.id ==
>> (getIterationRuntimeContext.getSuperstepNumber)
>> })
>> val residual_2 = workset filter {_.id != -1}
>> val sum = VectorDataSet.add(old_sum, current)
>>
>> (sum map (new RichMapFunction[Vector, Vector]{
>> def map(x: Vector) = new
>> Vector(getIterationRuntimeContext.getSuperstepNumber, x.values)
>> }),
>> residual_2 union sum)
>> }
>>
>> The error:
>>
>> org.apache.flink.compiler.CompilerException: Error: The step function
>> does not reference the solution set.
>> at
>> org.apache.flink.compiler.PactCompiler$GraphCreatingVisitor.postVisit(PactCompiler.java:868)
>> at
>> org.apache.flink.compiler.PactCompiler$GraphCreatingVisitor.postVisit(PactCompiler.java:622)
>> at
>> org.apache.flink.api.common.operators.DualInputOperator.accept(DualInputOperator.java:283)
>> at
>> org.apache.flink.api.common.operators.SingleInputOperator.accept(SingleInputOperator.java:202)
>> at
>> org.apache.flink.api.common.operators.GenericDataSinkBase.accept(GenericDataSinkBase.java:286)
>> at org.apache.flink.api.common.Plan.accept(Plan.java:281)
>> at org.apache.flink.compiler.PactCompiler.compile(PactCompiler.java:517)
>> at org.apache.flink.compiler.PactCompiler.compile(PactCompiler.java:466)
>> at
>> org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:196)
>> at
>> org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:209)
>> at org.apache.flink.client.program.Client.run(Client.java:285)
>> at org.apache.flink.client.program.Client.run(Client.java:230)
>> at
>> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:347)
>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:334)
>> at
>> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1001)
>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1025)
>>
>> Thanks!
>> Cheers,
>> Max
>>
>
>
Re: Forced to use Solution Set in Step Function
Posted by Stephan Ewen <se...@apache.org>.
Hey!
Is the algorithm you are using a delta iteration in fact. If you actually
do not use the solution set, can you model it as a bulk-iteration?
If you actually need the solution set to accumulate data, we can probably
deactivate that check in the compiler. As far as I remember, there is no
requirement in the runtime to join with the solution set. The check is
meant to help programmers that forgot the join...
Greetings,
Stephan
On Tue, Oct 7, 2014 at 3:13 PM, Maximilian Alber <alber.maximilian@gmail.com
> wrote:
> Hi Flinksters!
>
> I would like to use iterateDelta function. I don't need the solution set
> inside the step function, because I generate a different values out of the
> working set. Unfortunately the compiler of the development version doesn't
> like that. Is there a workaround?
>
> The code:
>
> val residual_2a = residual_2 union
> env.fromCollection(Seq(Vector.zeros(config.dimensions)))
> val emptyDataSet = env.fromCollection[Vector](Seq())
> val cumSum = emptyDataSet.iterateDelta(residual_2a, 1000000, Array("id")) {
> (solutionset, workset) =>
> val old_sum = workset filter {_.id == -1}
> val current = workset filter (new RichFilterFunction[Vector]{
> def filter(x: Vector) = x.id ==
> (getIterationRuntimeContext.getSuperstepNumber)
> })
> val residual_2 = workset filter {_.id != -1}
> val sum = VectorDataSet.add(old_sum, current)
>
> (sum map (new RichMapFunction[Vector, Vector]{
> def map(x: Vector) = new
> Vector(getIterationRuntimeContext.getSuperstepNumber, x.values)
> }),
> residual_2 union sum)
> }
>
> The error:
>
> org.apache.flink.compiler.CompilerException: Error: The step function does
> not reference the solution set.
> at
> org.apache.flink.compiler.PactCompiler$GraphCreatingVisitor.postVisit(PactCompiler.java:868)
> at
> org.apache.flink.compiler.PactCompiler$GraphCreatingVisitor.postVisit(PactCompiler.java:622)
> at
> org.apache.flink.api.common.operators.DualInputOperator.accept(DualInputOperator.java:283)
> at
> org.apache.flink.api.common.operators.SingleInputOperator.accept(SingleInputOperator.java:202)
> at
> org.apache.flink.api.common.operators.GenericDataSinkBase.accept(GenericDataSinkBase.java:286)
> at org.apache.flink.api.common.Plan.accept(Plan.java:281)
> at org.apache.flink.compiler.PactCompiler.compile(PactCompiler.java:517)
> at org.apache.flink.compiler.PactCompiler.compile(PactCompiler.java:466)
> at org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:196)
> at org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:209)
> at org.apache.flink.client.program.Client.run(Client.java:285)
> at org.apache.flink.client.program.Client.run(Client.java:230)
> at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:347)
> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:334)
> at
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1001)
> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1025)
>
> Thanks!
> Cheers,
> Max
>