You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by rbraeunlich <gi...@git.apache.org> on 2015/05/26 23:48:44 UTC

[GitHub] flink pull request: basic TfidfTransformer

GitHub user rbraeunlich opened a pull request:

    https://github.com/apache/flink/pull/730

    basic TfidfTransformer

    Hi everybody,
    
    due to [Flink-1999](https://issues.apache.org/jira/browse/FLINK-1999) we created a first implementation of a TfIdfTranformer.
    There is still one problem left, because using modulo after the hashing causes collisions.
    Nevertheless, we would be glad to receive some comments to our implementation.
    
    Cheers,
    Ronny

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/rbraeunlich/flink tfidf

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/730.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #730
    
----
commit 9e9ac219b619ddfbab4f616165d038900b7726db
Author: Ronny Bräunlich <r....@gmail.com>
Date:   2015-05-15T09:18:00Z

    create TfIdfTransformer

commit 42ef7c00a832e21d7391e1011031bda162d930f1
Author: Ronny Bräunlich <r....@gmail.com>
Date:   2015-05-16T14:38:28Z

    fix import in TfIdfTranformer and add first basic test case

commit 82385b764f45f955cd88590b7657467689d096ed
Author: Ronny Bräunlich <r....@gmail.com>
Date:   2015-05-15T09:18:00Z

    create TfIdfTransformer and add first basic test case

commit 7242728b1c24027203f1ff91476de9acb9bbf3a7
Author: diva1012 <vs...@gmail.com>
Date:   2015-05-17T11:42:40Z

    Changes merged
    
    Merge remote-tracking branch 'rbraeunlich/tfidf' into tfidf
    
    Conflicts:
    	flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala

commit 9c2c181624bb81f3ed83a4a774339251508644f1
Author: diva1012 <vs...@gmail.com>
Date:   2015-05-17T17:40:00Z

    Small fix of the test class. (The Sparse vector contains index -> value tuples, so we have to take only the value and not the whole tuple for the comparisson)

commit 8b17385e34b7f139a2649f80edc81744277fcfae
Author: diva1012 <vs...@gmail.com>
Date:   2015-05-18T06:41:58Z

    Word count implementation simplified.

commit 229fac5f835ce05dd03544f7dd7c0df7952f18e9
Author: diva1012 <vs...@gmail.com>
Date:   2015-05-18T11:35:43Z

    TF calculation fixed

commit e1ea4437e42860d8ed7820c32e08d7a2d1152b08
Author: diva1012 <vs...@gmail.com>
Date:   2015-05-19T20:44:31Z

    Transformer improved: now we get SparseVector for each document that contains all words.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by rbraeunlich <gi...@git.apache.org>.
Github user rbraeunlich commented on the pull request:

    https://github.com/apache/flink/pull/730#issuecomment-174934843
  
    We cannot add a Apache license to the test poems because they are public domain.
    I'm not sure how to solve this with the style checker.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: basic TfidfTransformer

Posted by aalexandrov <gi...@git.apache.org>.
Github user aalexandrov commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r31113442
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,107 @@
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.math.log
    +import scala.util.hashing.MurmurHash3
    +import scala.collection.mutable.LinkedHashSet;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents.
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val params = transformParameters.get(StopWordParameter)
    +
    +    // Here we will store the words in he form (docId, word, count)
    +    // Count represent the occurrence of "word" in document "docId"
    +    val wordCounts = input
    +      //count the words
    +      .flatMap(t => {
    +      //create tuples docId, word, 1
    +      t._2.map(s => (t._1, s, 1))
    +    })
    +      .filter(t => !transformParameters.apply(StopWordParameter).contains(t._2))
    +      //group by document and word
    +      .groupBy(0, 1)
    +      // calculate the occurrence count of each word in specific document
    +      .sum(2)
    +
    +    val dictionary = wordCounts
    +      .map(t => LinkedHashSet(t._2))
    +      .reduce((set1, set2) => set1 ++ set2)
    --- End diff --
    
    better use `groupReduce` which will give you the materialized set right away.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r32620542
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    --- End diff --
    
    using `"""(?u)\b\w\w+\b""".r` relieves you from escaping, I think


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r32623711
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    +          s2 => (t._1, s2.toLowerCase, 1)
    +          )
    +      })
    +    })
    +      .filter(t => !transformParameters.apply(StopWordParameter).contains(t._2))
    +      //group by document and word
    +      .groupBy(0, 1)
    +      // calculate the occurrence count of each word in specific document
    +      .sum(2)
    +
    +    val dictionary = matchedWordCounts
    +      .map(t => LinkedHashSet(t._2))
    +      .reduce((set1, set2) => set1 ++ set2)
    +      //to make sure we only create one dictionary
    +      .setParallelism(1)
    +      .map(set => set.zipWithIndex.toMap)
    +      .flatMap(m => m.toList)
    +
    +    val numberOfWords = matchedWordCounts
    +      .map(t => (t._2))
    +      .distinct(t => t)
    +      .map(t => 1)
    +      .reduce(_ + _);
    +
    +    val idf: DataSet[(String, Double)] = calculateIDF(matchedWordCounts)
    +    val tf: DataSet[(Int, String, Int)] = matchedWordCounts
    +
    +    // docId, word, tfIdf
    +    val tfIdf = tf.join(idf).where(1).equalTo(0) {
    +      (t1, t2) => (t1._1, t1._2, t1._3.toDouble * t2._2)
    +    }
    +
    +    val res = tfIdf.crossWithTiny(numberOfWords)
    +      // docId, word, tfIdf, numberOfWords
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._2))
    +      //assign every word its position
    +      .joinWithTiny(dictionary).where(1).equalTo(0)
    +      //join the tuples (docId, word, tfIdf, numberOfWords, index
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._1._4, t._2._2))
    +      .map(t => {
    +      (t._1, List[(Int, Double)]((t._5, t._3)), t._4)
    +    })
    +      .groupBy(t => t._1)
    +      .reduce((t1, t2) => (t1._1, t1._2 ++ t2._2, t1._3))
    +      .map(t => (t._1, SparseVector.fromCOO(t._3, t._2.toIterable)))
    +
    +    res
    +  }
    +
    +  /**
    +   * Takes the DataSet of DocId, Word and WordCount and calculates the IDF as tuple of word and idf
    +   * @param set
    +   */
    +  private def calculateIDF(set: DataSet[(Int, String, Int)]) = {
    +    val totalNumberOfDocuments = set
    +      //map the input only to the docId
    +      .map(t => t._1)
    +      .groupBy(i => i)
    +      //reduce to unique docIds
    +      .reduce((t1, t2) => t1)
    +      .count();
    +
    +    val idf = set
    --- End diff --
    
    you can remove the value assignment here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r32619450
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    --- End diff --
    
    We don't use author tags.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by pefi1011 <gi...@git.apache.org>.
Github user pefi1011 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r33425453
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    +          s2 => (t._1, s2.toLowerCase, 1)
    --- End diff --
    
    Done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r32623295
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    +          s2 => (t._1, s2.toLowerCase, 1)
    +          )
    +      })
    +    })
    +      .filter(t => !transformParameters.apply(StopWordParameter).contains(t._2))
    +      //group by document and word
    +      .groupBy(0, 1)
    +      // calculate the occurrence count of each word in specific document
    +      .sum(2)
    +
    +    val dictionary = matchedWordCounts
    +      .map(t => LinkedHashSet(t._2))
    +      .reduce((set1, set2) => set1 ++ set2)
    +      //to make sure we only create one dictionary
    +      .setParallelism(1)
    +      .map(set => set.zipWithIndex.toMap)
    +      .flatMap(m => m.toList)
    +
    +    val numberOfWords = matchedWordCounts
    +      .map(t => (t._2))
    +      .distinct(t => t)
    +      .map(t => 1)
    +      .reduce(_ + _);
    +
    +    val idf: DataSet[(String, Double)] = calculateIDF(matchedWordCounts)
    +    val tf: DataSet[(Int, String, Int)] = matchedWordCounts
    +
    +    // docId, word, tfIdf
    +    val tfIdf = tf.join(idf).where(1).equalTo(0) {
    +      (t1, t2) => (t1._1, t1._2, t1._3.toDouble * t2._2)
    +    }
    +
    +    val res = tfIdf.crossWithTiny(numberOfWords)
    +      // docId, word, tfIdf, numberOfWords
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._2))
    +      //assign every word its position
    +      .joinWithTiny(dictionary).where(1).equalTo(0)
    +      //join the tuples (docId, word, tfIdf, numberOfWords, index
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._1._4, t._2._2))
    +      .map(t => {
    +      (t._1, List[(Int, Double)]((t._5, t._3)), t._4)
    +    })
    +      .groupBy(t => t._1)
    +      .reduce((t1, t2) => (t1._1, t1._2 ++ t2._2, t1._3))
    +      .map(t => (t._1, SparseVector.fromCOO(t._3, t._2.toIterable)))
    +
    +    res
    +  }
    +
    +  /**
    +   * Takes the DataSet of DocId, Word and WordCount and calculates the IDF as tuple of word and idf
    +   * @param set
    +   */
    +  private def calculateIDF(set: DataSet[(Int, String, Int)]) = {
    +    val totalNumberOfDocuments = set
    +      //map the input only to the docId
    +      .map(t => t._1)
    +      .groupBy(i => i)
    +      //reduce to unique docIds
    +      .reduce((t1, t2) => t1)
    +      .count();
    --- End diff --
    
    `count` triggers the execution of the dataflow and returns the result to the client. It's better to keep the data as a dataset and use instead a broadcast variable to send it to subsequent operations.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by chiwanpark <gi...@git.apache.org>.
Github user chiwanpark commented on the pull request:

    https://github.com/apache/flink/pull/730#issuecomment-170877495
  
    Is there any updates? The tests are failing because of style checking of test data.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by pefi1011 <gi...@git.apache.org>.
Github user pefi1011 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r33424858
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    +          s2 => (t._1, s2.toLowerCase, 1)
    +          )
    +      })
    +    })
    +      .filter(t => !transformParameters.apply(StopWordParameter).contains(t._2))
    +      //group by document and word
    +      .groupBy(0, 1)
    +      // calculate the occurrence count of each word in specific document
    +      .sum(2)
    +
    +    val dictionary = matchedWordCounts
    +      .map(t => LinkedHashSet(t._2))
    +      .reduce((set1, set2) => set1 ++ set2)
    +      //to make sure we only create one dictionary
    +      .setParallelism(1)
    +      .map(set => set.zipWithIndex.toMap)
    +      .flatMap(m => m.toList)
    --- End diff --
    
    Done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by pefi1011 <gi...@git.apache.org>.
Github user pefi1011 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r33425438
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    +          s2 => (t._1, s2.toLowerCase, 1)
    +          )
    +      })
    +    })
    +      .filter(t => !transformParameters.apply(StopWordParameter).contains(t._2))
    +      //group by document and word
    +      .groupBy(0, 1)
    +      // calculate the occurrence count of each word in specific document
    +      .sum(2)
    +
    +    val dictionary = matchedWordCounts
    +      .map(t => LinkedHashSet(t._2))
    +      .reduce((set1, set2) => set1 ++ set2)
    +      //to make sure we only create one dictionary
    +      .setParallelism(1)
    +      .map(set => set.zipWithIndex.toMap)
    +      .flatMap(m => m.toList)
    +
    +    val numberOfWords = matchedWordCounts
    +      .map(t => (t._2))
    +      .distinct(t => t)
    +      .map(t => 1)
    +      .reduce(_ + _);
    +
    +    val idf: DataSet[(String, Double)] = calculateIDF(matchedWordCounts)
    +    val tf: DataSet[(Int, String, Int)] = matchedWordCounts
    +
    +    // docId, word, tfIdf
    +    val tfIdf = tf.join(idf).where(1).equalTo(0) {
    +      (t1, t2) => (t1._1, t1._2, t1._3.toDouble * t2._2)
    +    }
    +
    +    val res = tfIdf.crossWithTiny(numberOfWords)
    --- End diff --
    
    Done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by pefi1011 <gi...@git.apache.org>.
Github user pefi1011 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r33426025
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    +          s2 => (t._1, s2.toLowerCase, 1)
    +          )
    +      })
    +    })
    +      .filter(t => !transformParameters.apply(StopWordParameter).contains(t._2))
    +      //group by document and word
    +      .groupBy(0, 1)
    +      // calculate the occurrence count of each word in specific document
    +      .sum(2)
    +
    +    val dictionary = matchedWordCounts
    +      .map(t => LinkedHashSet(t._2))
    +      .reduce((set1, set2) => set1 ++ set2)
    +      //to make sure we only create one dictionary
    +      .setParallelism(1)
    +      .map(set => set.zipWithIndex.toMap)
    +      .flatMap(m => m.toList)
    +
    +    val numberOfWords = matchedWordCounts
    +      .map(t => (t._2))
    +      .distinct(t => t)
    +      .map(t => 1)
    +      .reduce(_ + _);
    +
    +    val idf: DataSet[(String, Double)] = calculateIDF(matchedWordCounts)
    +    val tf: DataSet[(Int, String, Int)] = matchedWordCounts
    +
    +    // docId, word, tfIdf
    +    val tfIdf = tf.join(idf).where(1).equalTo(0) {
    +      (t1, t2) => (t1._1, t1._2, t1._3.toDouble * t2._2)
    +    }
    +
    +    val res = tfIdf.crossWithTiny(numberOfWords)
    +      // docId, word, tfIdf, numberOfWords
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._2))
    +      //assign every word its position
    +      .joinWithTiny(dictionary).where(1).equalTo(0)
    +      //join the tuples (docId, word, tfIdf, numberOfWords, index
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._1._4, t._2._2))
    +      .map(t => {
    +      (t._1, List[(Int, Double)]((t._5, t._3)), t._4)
    +    })
    +      .groupBy(t => t._1)
    +      .reduce((t1, t2) => (t1._1, t1._2 ++ t2._2, t1._3))
    +      .map(t => (t._1, SparseVector.fromCOO(t._3, t._2.toIterable)))
    +
    +    res
    +  }
    +
    +  /**
    +   * Takes the DataSet of DocId, Word and WordCount and calculates the IDF as tuple of word and idf
    +   * @param set
    +   */
    +  private def calculateIDF(set: DataSet[(Int, String, Int)]) = {
    +    val totalNumberOfDocuments = set
    +      //map the input only to the docId
    +      .map(t => t._1)
    +      .groupBy(i => i)
    +      //reduce to unique docIds
    +      .reduce((t1, t2) => t1)
    +      .count();
    +
    +    val idf = set
    +      //for tuple docId, Word and wordcount only take the word and a 1
    +      .map(t => (t._2, 1))
    +      //group by word
    +      .groupBy(t => t._1)
    +      //and count the documents
    +      .reduce((t1, t2) => (t1._1, t1._2 + t2._2))
    +      //calculate IDF
    +      .map(t => (t._1, log(totalNumberOfDocuments.toDouble / t._2.toDouble) + 1.0))
    +    idf
    +  }
    +}
    +
    +object StopWordParameter extends Parameter[Set[String]] with Serializable {
    --- End diff --
    
    @tillrohrmann  how to do that?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by pefi1011 <gi...@git.apache.org>.
Github user pefi1011 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r33425463
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    --- End diff --
    
    Done



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r32620440
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    +          s2 => (t._1, s2.toLowerCase, 1)
    --- End diff --
    
    Maybe replace `s` by `textLine` and `s2` by `matchedWord`. This eases the understanding.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by rbraeunlich <gi...@git.apache.org>.
Github user rbraeunlich commented on the pull request:

    https://github.com/apache/flink/pull/730#issuecomment-174997159
  
    We used scikit-learn as a reference.
    
    
    Am 26.01.2016 um 21:36 schrieb Till Rohrmann <no...@github.com>:
    
    > How have you done it in the first place for the poems? Don't you have an
    > automatic tool for that?
    > 
    > On Tue, Jan 26, 2016 at 1:21 PM, Ronny Bräunlich <no...@github.com>
    > wrote:
    > 
    > > Somebody has to do the new calculations. I suppose that'll be the biggest
    > > amount of work.
    > >
    > > —
    > > Reply to this email directly or view it on GitHub
    > > <https://github.com/apache/flink/pull/730#issuecomment-174979030>.
    > >
    > —
    > Reply to this email directly or view it on GitHub.
    > 
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the pull request:

    https://github.com/apache/flink/pull/730#issuecomment-112824195
  
    Thanks for your contribution @rbraeunlich et al. It looks very good. I had some minor comments which you find in the code. While going through the code and looking at scikit-learn I was wondering whether we shouldn't separate the IDF from the TF part and realize them as two `Transformer`.
    
    The TF transformer could be something like scikit-learn's `CountVectorizer` or a the HashingTermFrequency transformer. The HashingTermFrequency transformer could look like the code you've written for calculating the `dictionary`. This tranformer will take care of generating the numeric vector representing the term frequency.
    
    The IDF transformer is then trained on the numeric vectors generated from the corpus. The training means calculating the IDF values for the different terms. You can then predict new documents by first transforming the document using the TF transformer (giving a vector of term frequencies) and then apply the IDF weighting.
    
    What do you think?
    
    We also updated the pipelining mechanism. Thus, it would be great if you could rebase your PR on the latest master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r32623073
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    +          s2 => (t._1, s2.toLowerCase, 1)
    +          )
    +      })
    +    })
    +      .filter(t => !transformParameters.apply(StopWordParameter).contains(t._2))
    +      //group by document and word
    +      .groupBy(0, 1)
    +      // calculate the occurrence count of each word in specific document
    +      .sum(2)
    +
    +    val dictionary = matchedWordCounts
    +      .map(t => LinkedHashSet(t._2))
    +      .reduce((set1, set2) => set1 ++ set2)
    +      //to make sure we only create one dictionary
    +      .setParallelism(1)
    --- End diff --
    
    You don't have to do this for an all reduce operation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r32624099
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    +          s2 => (t._1, s2.toLowerCase, 1)
    +          )
    +      })
    +    })
    +      .filter(t => !transformParameters.apply(StopWordParameter).contains(t._2))
    +      //group by document and word
    +      .groupBy(0, 1)
    +      // calculate the occurrence count of each word in specific document
    +      .sum(2)
    +
    +    val dictionary = matchedWordCounts
    +      .map(t => LinkedHashSet(t._2))
    +      .reduce((set1, set2) => set1 ++ set2)
    +      //to make sure we only create one dictionary
    +      .setParallelism(1)
    +      .map(set => set.zipWithIndex.toMap)
    +      .flatMap(m => m.toList)
    +
    +    val numberOfWords = matchedWordCounts
    +      .map(t => (t._2))
    +      .distinct(t => t)
    +      .map(t => 1)
    +      .reduce(_ + _);
    +
    +    val idf: DataSet[(String, Double)] = calculateIDF(matchedWordCounts)
    +    val tf: DataSet[(Int, String, Int)] = matchedWordCounts
    +
    +    // docId, word, tfIdf
    +    val tfIdf = tf.join(idf).where(1).equalTo(0) {
    +      (t1, t2) => (t1._1, t1._2, t1._3.toDouble * t2._2)
    +    }
    +
    +    val res = tfIdf.crossWithTiny(numberOfWords)
    +      // docId, word, tfIdf, numberOfWords
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._2))
    +      //assign every word its position
    +      .joinWithTiny(dictionary).where(1).equalTo(0)
    +      //join the tuples (docId, word, tfIdf, numberOfWords, index
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._1._4, t._2._2))
    --- End diff --
    
    double nested tuples get quite confusing. Maybe you can name the different fields.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r32628581
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    --- End diff --
    
    Ah ok, I see. But it's written that this is strongly discouraged. Why not using the greedy/lazy modifiers?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r32619716
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    +          s2 => (t._1, s2.toLowerCase, 1)
    +          )
    +      })
    +    })
    +      .filter(t => !transformParameters.apply(StopWordParameter).contains(t._2))
    +      //group by document and word
    +      .groupBy(0, 1)
    +      // calculate the occurrence count of each word in specific document
    +      .sum(2)
    +
    +    val dictionary = matchedWordCounts
    +      .map(t => LinkedHashSet(t._2))
    +      .reduce((set1, set2) => set1 ++ set2)
    +      //to make sure we only create one dictionary
    +      .setParallelism(1)
    +      .map(set => set.zipWithIndex.toMap)
    +      .flatMap(m => m.toList)
    +
    +    val numberOfWords = matchedWordCounts
    +      .map(t => (t._2))
    +      .distinct(t => t)
    +      .map(t => 1)
    +      .reduce(_ + _);
    +
    +    val idf: DataSet[(String, Double)] = calculateIDF(matchedWordCounts)
    +    val tf: DataSet[(Int, String, Int)] = matchedWordCounts
    +
    +    // docId, word, tfIdf
    +    val tfIdf = tf.join(idf).where(1).equalTo(0) {
    +      (t1, t2) => (t1._1, t1._2, t1._3.toDouble * t2._2)
    +    }
    +
    +    val res = tfIdf.crossWithTiny(numberOfWords)
    +      // docId, word, tfIdf, numberOfWords
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._2))
    +      //assign every word its position
    +      .joinWithTiny(dictionary).where(1).equalTo(0)
    +      //join the tuples (docId, word, tfIdf, numberOfWords, index
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._1._4, t._2._2))
    +      .map(t => {
    +      (t._1, List[(Int, Double)]((t._5, t._3)), t._4)
    +    })
    +      .groupBy(t => t._1)
    +      .reduce((t1, t2) => (t1._1, t1._2 ++ t2._2, t1._3))
    +      .map(t => (t._1, SparseVector.fromCOO(t._3, t._2.toIterable)))
    +
    +    res
    +  }
    +
    +  /**
    +   * Takes the DataSet of DocId, Word and WordCount and calculates the IDF as tuple of word and idf
    +   * @param set
    +   */
    +  private def calculateIDF(set: DataSet[(Int, String, Int)]) = {
    +    val totalNumberOfDocuments = set
    +      //map the input only to the docId
    +      .map(t => t._1)
    +      .groupBy(i => i)
    +      //reduce to unique docIds
    +      .reduce((t1, t2) => t1)
    +      .count();
    +
    +    val idf = set
    +      //for tuple docId, Word and wordcount only take the word and a 1
    +      .map(t => (t._2, 1))
    +      //group by word
    +      .groupBy(t => t._1)
    +      //and count the documents
    +      .reduce((t1, t2) => (t1._1, t1._2 + t2._2))
    +      //calculate IDF
    +      .map(t => (t._1, log(totalNumberOfDocuments.toDouble / t._2.toDouble) + 1.0))
    +    idf
    +  }
    +}
    +
    +object StopWordParameter extends Parameter[Set[String]] with Serializable {
    --- End diff --
    
    Parameters should be defined within the scope of an companion object.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r32624606
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    --- End diff --
    
    Why do you have a `Seq[String]` as document type? Is `String` not sufficient?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: basic TfidfTransformer

Posted by aalexandrov <gi...@git.apache.org>.
Github user aalexandrov commented on the pull request:

    https://github.com/apache/flink/pull/730#issuecomment-106910157
  
    @rbraeunlich please squash the commits and prepare this for merging.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the pull request:

    https://github.com/apache/flink/pull/730#issuecomment-174988478
  
    How have you done it in the first place for the poems? Don't you have an
    automatic tool for that?
    
    On Tue, Jan 26, 2016 at 1:21 PM, Ronny Bräunlich <no...@github.com>
    wrote:
    
    > Somebody has to do the new calculations. I suppose that'll be the biggest
    > amount of work.
    >
    > —
    > Reply to this email directly or view it on GitHub
    > <https://github.com/apache/flink/pull/730#issuecomment-174979030>.
    >



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by pefi1011 <gi...@git.apache.org>.
Github user pefi1011 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r33425478
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    --- End diff --
    
    Done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the pull request:

    https://github.com/apache/flink/pull/730#issuecomment-174955141
  
    What do you mean with rewriting the tests? It would mean to re-calculate
    the tfidf values, right? Would this mean a lot of work?
    
    On Tue, Jan 26, 2016 at 11:33 AM, Ronny Bräunlich <no...@github.com>
    wrote:
    
    > Well, writing our own poems would result in rewriting the tests...
    > Does the checkstyle plugin maybe expect public domain header in a certain
    > format?
    >
    > —
    > Reply to this email directly or view it on GitHub
    > <https://github.com/apache/flink/pull/730#issuecomment-174947692>.
    >



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by pefi1011 <gi...@git.apache.org>.
Github user pefi1011 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r33424859
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    +          s2 => (t._1, s2.toLowerCase, 1)
    +          )
    +      })
    +    })
    +      .filter(t => !transformParameters.apply(StopWordParameter).contains(t._2))
    +      //group by document and word
    +      .groupBy(0, 1)
    +      // calculate the occurrence count of each word in specific document
    +      .sum(2)
    +
    +    val dictionary = matchedWordCounts
    +      .map(t => LinkedHashSet(t._2))
    +      .reduce((set1, set2) => set1 ++ set2)
    +      //to make sure we only create one dictionary
    +      .setParallelism(1)
    +      .map(set => set.zipWithIndex.toMap)
    +      .flatMap(m => m.toList)
    +
    +    val numberOfWords = matchedWordCounts
    +      .map(t => (t._2))
    +      .distinct(t => t)
    +      .map(t => 1)
    +      .reduce(_ + _);
    +
    +    val idf: DataSet[(String, Double)] = calculateIDF(matchedWordCounts)
    +    val tf: DataSet[(Int, String, Int)] = matchedWordCounts
    +
    +    // docId, word, tfIdf
    +    val tfIdf = tf.join(idf).where(1).equalTo(0) {
    +      (t1, t2) => (t1._1, t1._2, t1._3.toDouble * t2._2)
    +    }
    +
    +    val res = tfIdf.crossWithTiny(numberOfWords)
    +      // docId, word, tfIdf, numberOfWords
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._2))
    +      //assign every word its position
    +      .joinWithTiny(dictionary).where(1).equalTo(0)
    +      //join the tuples (docId, word, tfIdf, numberOfWords, index
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._1._4, t._2._2))
    +      .map(t => {
    +      (t._1, List[(Int, Double)]((t._5, t._3)), t._4)
    +    })
    +      .groupBy(t => t._1)
    +      .reduce((t1, t2) => (t1._1, t1._2 ++ t2._2, t1._3))
    +      .map(t => (t._1, SparseVector.fromCOO(t._3, t._2.toIterable)))
    +
    +    res
    +  }
    +
    +  /**
    +   * Takes the DataSet of DocId, Word and WordCount and calculates the IDF as tuple of word and idf
    +   * @param set
    +   */
    +  private def calculateIDF(set: DataSet[(Int, String, Int)]) = {
    --- End diff --
    
    Done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r32622586
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    --- End diff --
    
    What does the `(?u)` do?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by pefi1011 <gi...@git.apache.org>.
Github user pefi1011 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r33422802
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    +          s2 => (t._1, s2.toLowerCase, 1)
    +          )
    +      })
    +    })
    +      .filter(t => !transformParameters.apply(StopWordParameter).contains(t._2))
    +      //group by document and word
    +      .groupBy(0, 1)
    +      // calculate the occurrence count of each word in specific document
    +      .sum(2)
    +
    +    val dictionary = matchedWordCounts
    +      .map(t => LinkedHashSet(t._2))
    +      .reduce((set1, set2) => set1 ++ set2)
    +      //to make sure we only create one dictionary
    +      .setParallelism(1)
    +      .map(set => set.zipWithIndex.toMap)
    +      .flatMap(m => m.toList)
    +
    +    val numberOfWords = matchedWordCounts
    +      .map(t => (t._2))
    +      .distinct(t => t)
    +      .map(t => 1)
    +      .reduce(_ + _);
    +
    +    val idf: DataSet[(String, Double)] = calculateIDF(matchedWordCounts)
    +    val tf: DataSet[(Int, String, Int)] = matchedWordCounts
    +
    +    // docId, word, tfIdf
    +    val tfIdf = tf.join(idf).where(1).equalTo(0) {
    +      (t1, t2) => (t1._1, t1._2, t1._3.toDouble * t2._2)
    +    }
    +
    +    val res = tfIdf.crossWithTiny(numberOfWords)
    +      // docId, word, tfIdf, numberOfWords
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._2))
    +      //assign every word its position
    +      .joinWithTiny(dictionary).where(1).equalTo(0)
    +      //join the tuples (docId, word, tfIdf, numberOfWords, index
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._1._4, t._2._2))
    +      .map(t => {
    +      (t._1, List[(Int, Double)]((t._5, t._3)), t._4)
    +    })
    +      .groupBy(t => t._1)
    +      .reduce((t1, t2) => (t1._1, t1._2 ++ t2._2, t1._3))
    +      .map(t => (t._1, SparseVector.fromCOO(t._3, t._2.toIterable)))
    +
    +    res
    +  }
    +
    +  /**
    +   * Takes the DataSet of DocId, Word and WordCount and calculates the IDF as tuple of word and idf
    +   * @param set
    +   */
    +  private def calculateIDF(set: DataSet[(Int, String, Int)]) = {
    +    val totalNumberOfDocuments = set
    +      //map the input only to the docId
    +      .map(t => t._1)
    +      .groupBy(i => i)
    +      //reduce to unique docIds
    +      .reduce((t1, t2) => t1)
    --- End diff --
    
    Distinct is not possible because the result of the map is not not a tuple type or we don't know how to use it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r32623851
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    +          s2 => (t._1, s2.toLowerCase, 1)
    +          )
    +      })
    +    })
    +      .filter(t => !transformParameters.apply(StopWordParameter).contains(t._2))
    +      //group by document and word
    +      .groupBy(0, 1)
    +      // calculate the occurrence count of each word in specific document
    +      .sum(2)
    +
    +    val dictionary = matchedWordCounts
    +      .map(t => LinkedHashSet(t._2))
    +      .reduce((set1, set2) => set1 ++ set2)
    +      //to make sure we only create one dictionary
    +      .setParallelism(1)
    +      .map(set => set.zipWithIndex.toMap)
    +      .flatMap(m => m.toList)
    +
    +    val numberOfWords = matchedWordCounts
    +      .map(t => (t._2))
    +      .distinct(t => t)
    +      .map(t => 1)
    +      .reduce(_ + _);
    +
    +    val idf: DataSet[(String, Double)] = calculateIDF(matchedWordCounts)
    +    val tf: DataSet[(Int, String, Int)] = matchedWordCounts
    +
    +    // docId, word, tfIdf
    +    val tfIdf = tf.join(idf).where(1).equalTo(0) {
    +      (t1, t2) => (t1._1, t1._2, t1._3.toDouble * t2._2)
    +    }
    +
    +    val res = tfIdf.crossWithTiny(numberOfWords)
    --- End diff --
    
    better to use a broadcast variable here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r32620238
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    --- End diff --
    
    why this formula (`+ 1`)? What happens if `documents that contain the word = 0`? Any reference paper?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the pull request:

    https://github.com/apache/flink/pull/730#issuecomment-174977804
  
    Would this involve too much work?
    
    On Tue, Jan 26, 2016 at 12:03 PM, Ronny Bräunlich <no...@github.com>
    wrote:
    
    > Well, one test would have to be changed. That test currently reads the
    > values from poems_out.txt, orders them and compares them afterwards with
    > the calculated numbers (again ordered).
    >
    > —
    > Reply to this email directly or view it on GitHub
    > <https://github.com/apache/flink/pull/730#issuecomment-174957852>.
    >



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by aalexandrov <gi...@git.apache.org>.
Github user aalexandrov commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r32620391
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    --- End diff --
    
    I think this was aligned with the behaviour observed in skikit-learn.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by rbraeunlich <gi...@git.apache.org>.
Github user rbraeunlich commented on the pull request:

    https://github.com/apache/flink/pull/730#issuecomment-174979030
  
    Somebody has to do the new calculations. I suppose that'll be the biggest amount of work.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by rbraeunlich <gi...@git.apache.org>.
Github user rbraeunlich commented on the pull request:

    https://github.com/apache/flink/pull/730#issuecomment-174947692
  
    Well, writing our own poems would result in rewriting the tests...
    Does the checkstyle plugin maybe expect public domain header in a certain format?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the pull request:

    https://github.com/apache/flink/pull/730#issuecomment-112776948
  
    Maven verify does not complete. There are apparently files which don't contain the Apache license header. This failure should be visible when you run `mvn clean verify` locally.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by pefi1011 <gi...@git.apache.org>.
Github user pefi1011 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r33424889
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    +          s2 => (t._1, s2.toLowerCase, 1)
    +          )
    +      })
    +    })
    +      .filter(t => !transformParameters.apply(StopWordParameter).contains(t._2))
    +      //group by document and word
    +      .groupBy(0, 1)
    +      // calculate the occurrence count of each word in specific document
    +      .sum(2)
    +
    +    val dictionary = matchedWordCounts
    +      .map(t => LinkedHashSet(t._2))
    +      .reduce((set1, set2) => set1 ++ set2)
    +      //to make sure we only create one dictionary
    +      .setParallelism(1)
    +      .map(set => set.zipWithIndex.toMap)
    +      .flatMap(m => m.toList)
    +
    +    val numberOfWords = matchedWordCounts
    +      .map(t => (t._2))
    +      .distinct(t => t)
    +      .map(t => 1)
    +      .reduce(_ + _);
    +
    +    val idf: DataSet[(String, Double)] = calculateIDF(matchedWordCounts)
    +    val tf: DataSet[(Int, String, Int)] = matchedWordCounts
    +
    +    // docId, word, tfIdf
    +    val tfIdf = tf.join(idf).where(1).equalTo(0) {
    +      (t1, t2) => (t1._1, t1._2, t1._3.toDouble * t2._2)
    +    }
    +
    +    val res = tfIdf.crossWithTiny(numberOfWords)
    +      // docId, word, tfIdf, numberOfWords
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._2))
    +      //assign every word its position
    +      .joinWithTiny(dictionary).where(1).equalTo(0)
    +      //join the tuples (docId, word, tfIdf, numberOfWords, index
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._1._4, t._2._2))
    +      .map(t => {
    --- End diff --
    
    Done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by rbraeunlich <gi...@git.apache.org>.
Github user rbraeunlich commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r33551797
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    --- End diff --
    
    We had some discussion about the type in [FLINK-1999](https://issues.apache.org/jira/browse/FLINK-1999) and since there were no objections we sticked to Seq[String]. With Seq[String] the caller has to decide how to split his text.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the pull request:

    https://github.com/apache/flink/pull/730#issuecomment-174940437
  
    Hmm according to http://www.apache.org/legal/resolved.html#can-works-placed-in-the-public-domain-be-included-in-apache-products, it should be no problem to include work from the public domain in ASF products. They only have to be attributed accordingly. What we then would have to do is to simply exclude them from the checkstyle check.
    
    Alternatively, we could also write/generate our own poems which we put under the ASL.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by rbraeunlich <gi...@git.apache.org>.
Github user rbraeunlich commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r32623957
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    --- End diff --
    
    That's a modifier to make the regex non-greedy, e.g. see [here](http://www.regular-expressions.info/modifiers.html).
    Ignoring single characters was intended, since scikit, as reference, does the same.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: basic TfidfTransformer

Posted by aalexandrov <gi...@git.apache.org>.
Github user aalexandrov commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r31114073
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,107 @@
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.math.log
    +import scala.util.hashing.MurmurHash3
    +import scala.collection.mutable.LinkedHashSet;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents.
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val params = transformParameters.get(StopWordParameter)
    +
    +    // Here we will store the words in he form (docId, word, count)
    +    // Count represent the occurrence of "word" in document "docId"
    +    val wordCounts = input
    +      //count the words
    +      .flatMap(t => {
    +      //create tuples docId, word, 1
    +      t._2.map(s => (t._1, s, 1))
    +    })
    +      .filter(t => !transformParameters.apply(StopWordParameter).contains(t._2))
    +      //group by document and word
    +      .groupBy(0, 1)
    +      // calculate the occurrence count of each word in specific document
    +      .sum(2)
    +
    +    val dictionary = wordCounts
    +      .map(t => LinkedHashSet(t._2))
    +      .reduce((set1, set2) => set1 ++ set2)
    +      .map(set => set.zipWithIndex.toMap)
    +      .flatMap(m => m.toList)
    +
    +    val numberOfWords = wordCounts
    +      .map(t => (t._2))
    +      .distinct(t => t)
    +      .map(t => 1)
    +      .reduce(_ + _);
    +
    +    val idf: DataSet[(String, Double)] = calculateIDF(wordCounts)
    +    val tf: DataSet[(Int, String, Int)] = wordCounts
    +
    +    // docId, word, tfIdf
    +    val tfIdf = tf.join(idf).where(1).equalTo(0) {
    +      (t1, t2) => (t1._1, t1._2, t1._3.toDouble * t2._2)
    +    }
    +
    +    val res = tfIdf.crossWithTiny(numberOfWords)
    +      // docId, word, tfIdf, numberOfWords
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._2))
    +      //assign every word its position
    +      .joinWithHuge(dictionary).where(1).equalTo(0)
    --- End diff --
    
    Use `.joinWithTiny` or `.join` or a broadcast variable.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by pefi1011 <gi...@git.apache.org>.
Github user pefi1011 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r33424878
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    +          s2 => (t._1, s2.toLowerCase, 1)
    +          )
    +      })
    +    })
    +      .filter(t => !transformParameters.apply(StopWordParameter).contains(t._2))
    +      //group by document and word
    +      .groupBy(0, 1)
    +      // calculate the occurrence count of each word in specific document
    +      .sum(2)
    +
    +    val dictionary = matchedWordCounts
    +      .map(t => LinkedHashSet(t._2))
    +      .reduce((set1, set2) => set1 ++ set2)
    +      //to make sure we only create one dictionary
    +      .setParallelism(1)
    +      .map(set => set.zipWithIndex.toMap)
    +      .flatMap(m => m.toList)
    +
    +    val numberOfWords = matchedWordCounts
    +      .map(t => (t._2))
    +      .distinct(t => t)
    +      .map(t => 1)
    +      .reduce(_ + _);
    +
    +    val idf: DataSet[(String, Double)] = calculateIDF(matchedWordCounts)
    +    val tf: DataSet[(Int, String, Int)] = matchedWordCounts
    +
    +    // docId, word, tfIdf
    +    val tfIdf = tf.join(idf).where(1).equalTo(0) {
    +      (t1, t2) => (t1._1, t1._2, t1._3.toDouble * t2._2)
    +    }
    +
    +    val res = tfIdf.crossWithTiny(numberOfWords)
    +      // docId, word, tfIdf, numberOfWords
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._2))
    +      //assign every word its position
    +      .joinWithTiny(dictionary).where(1).equalTo(0)
    +      //join the tuples (docId, word, tfIdf, numberOfWords, index
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._1._4, t._2._2))
    +      .map(t => {
    +      (t._1, List[(Int, Double)]((t._5, t._3)), t._4)
    +    })
    +      .groupBy(t => t._1)
    +      .reduce((t1, t2) => (t1._1, t1._2 ++ t2._2, t1._3))
    +      .map(t => (t._1, SparseVector.fromCOO(t._3, t._2.toIterable)))
    +
    +    res
    +  }
    +
    +  /**
    +   * Takes the DataSet of DocId, Word and WordCount and calculates the IDF as tuple of word and idf
    +   * @param set
    +   */
    +  private def calculateIDF(set: DataSet[(Int, String, Int)]) = {
    +    val totalNumberOfDocuments = set
    +      //map the input only to the docId
    +      .map(t => t._1)
    +      .groupBy(i => i)
    +      //reduce to unique docIds
    +      .reduce((t1, t2) => t1)
    +      .count();
    +
    +    val idf = set
    --- End diff --
    
    Done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r32623461
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    +          s2 => (t._1, s2.toLowerCase, 1)
    +          )
    +      })
    +    })
    +      .filter(t => !transformParameters.apply(StopWordParameter).contains(t._2))
    +      //group by document and word
    +      .groupBy(0, 1)
    +      // calculate the occurrence count of each word in specific document
    +      .sum(2)
    +
    +    val dictionary = matchedWordCounts
    +      .map(t => LinkedHashSet(t._2))
    +      .reduce((set1, set2) => set1 ++ set2)
    +      //to make sure we only create one dictionary
    +      .setParallelism(1)
    +      .map(set => set.zipWithIndex.toMap)
    +      .flatMap(m => m.toList)
    +
    +    val numberOfWords = matchedWordCounts
    +      .map(t => (t._2))
    +      .distinct(t => t)
    +      .map(t => 1)
    +      .reduce(_ + _);
    +
    +    val idf: DataSet[(String, Double)] = calculateIDF(matchedWordCounts)
    +    val tf: DataSet[(Int, String, Int)] = matchedWordCounts
    +
    +    // docId, word, tfIdf
    +    val tfIdf = tf.join(idf).where(1).equalTo(0) {
    +      (t1, t2) => (t1._1, t1._2, t1._3.toDouble * t2._2)
    +    }
    +
    +    val res = tfIdf.crossWithTiny(numberOfWords)
    +      // docId, word, tfIdf, numberOfWords
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._2))
    +      //assign every word its position
    +      .joinWithTiny(dictionary).where(1).equalTo(0)
    +      //join the tuples (docId, word, tfIdf, numberOfWords, index
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._1._4, t._2._2))
    +      .map(t => {
    +      (t._1, List[(Int, Double)]((t._5, t._3)), t._4)
    +    })
    +      .groupBy(t => t._1)
    +      .reduce((t1, t2) => (t1._1, t1._2 ++ t2._2, t1._3))
    +      .map(t => (t._1, SparseVector.fromCOO(t._3, t._2.toIterable)))
    +
    +    res
    +  }
    +
    +  /**
    +   * Takes the DataSet of DocId, Word and WordCount and calculates the IDF as tuple of word and idf
    +   * @param set
    +   */
    +  private def calculateIDF(set: DataSet[(Int, String, Int)]) = {
    +    val totalNumberOfDocuments = set
    +      //map the input only to the docId
    +      .map(t => t._1)
    +      .groupBy(i => i)
    +      //reduce to unique docIds
    +      .reduce((t1, t2) => t1)
    --- End diff --
    
    Why not using `distinct` here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r32620153
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    --- End diff --
    
    frequency


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the pull request:

    https://github.com/apache/flink/pull/730#issuecomment-112824285
  
    No problem, ping me when you've updated the PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r32624395
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    +          s2 => (t._1, s2.toLowerCase, 1)
    +          )
    +      })
    +    })
    +      .filter(t => !transformParameters.apply(StopWordParameter).contains(t._2))
    +      //group by document and word
    +      .groupBy(0, 1)
    +      // calculate the occurrence count of each word in specific document
    +      .sum(2)
    +
    +    val dictionary = matchedWordCounts
    +      .map(t => LinkedHashSet(t._2))
    +      .reduce((set1, set2) => set1 ++ set2)
    +      //to make sure we only create one dictionary
    +      .setParallelism(1)
    +      .map(set => set.zipWithIndex.toMap)
    +      .flatMap(m => m.toList)
    +
    +    val numberOfWords = matchedWordCounts
    +      .map(t => (t._2))
    +      .distinct(t => t)
    +      .map(t => 1)
    +      .reduce(_ + _);
    +
    +    val idf: DataSet[(String, Double)] = calculateIDF(matchedWordCounts)
    +    val tf: DataSet[(Int, String, Int)] = matchedWordCounts
    +
    +    // docId, word, tfIdf
    +    val tfIdf = tf.join(idf).where(1).equalTo(0) {
    +      (t1, t2) => (t1._1, t1._2, t1._3.toDouble * t2._2)
    +    }
    +
    +    val res = tfIdf.crossWithTiny(numberOfWords)
    +      // docId, word, tfIdf, numberOfWords
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._2))
    +      //assign every word its position
    +      .joinWithTiny(dictionary).where(1).equalTo(0)
    +      //join the tuples (docId, word, tfIdf, numberOfWords, index
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._1._4, t._2._2))
    +      .map(t => {
    --- End diff --
    
    Maybe some variable names with more than one character could be helpful.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by pefi1011 <gi...@git.apache.org>.
Github user pefi1011 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r33423401
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    +          s2 => (t._1, s2.toLowerCase, 1)
    +          )
    +      })
    +    })
    +      .filter(t => !transformParameters.apply(StopWordParameter).contains(t._2))
    +      //group by document and word
    +      .groupBy(0, 1)
    +      // calculate the occurrence count of each word in specific document
    +      .sum(2)
    +
    +    val dictionary = matchedWordCounts
    +      .map(t => LinkedHashSet(t._2))
    +      .reduce((set1, set2) => set1 ++ set2)
    +      //to make sure we only create one dictionary
    +      .setParallelism(1)
    +      .map(set => set.zipWithIndex.toMap)
    +      .flatMap(m => m.toList)
    +
    +    val numberOfWords = matchedWordCounts
    +      .map(t => (t._2))
    +      .distinct(t => t)
    +      .map(t => 1)
    +      .reduce(_ + _);
    +
    +    val idf: DataSet[(String, Double)] = calculateIDF(matchedWordCounts)
    +    val tf: DataSet[(Int, String, Int)] = matchedWordCounts
    +
    +    // docId, word, tfIdf
    +    val tfIdf = tf.join(idf).where(1).equalTo(0) {
    +      (t1, t2) => (t1._1, t1._2, t1._3.toDouble * t2._2)
    +    }
    +
    +    val res = tfIdf.crossWithTiny(numberOfWords)
    +      // docId, word, tfIdf, numberOfWords
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._2))
    +      //assign every word its position
    +      .joinWithTiny(dictionary).where(1).equalTo(0)
    +      //join the tuples (docId, word, tfIdf, numberOfWords, index
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._1._4, t._2._2))
    --- End diff --
    
    We are not sure how to name the parameters. We tried to use ".withForwardedFields" but we couldn't figure out how to use it. Could you please provide us an example? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r32627835
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    --- End diff --
    
    Fair enough. This should be documented, though. E.g. Adding 1 to the idf makes sure that terms with a zero idf get not completely ignored.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: basic TfidfTransformer

Posted by aalexandrov <gi...@git.apache.org>.
Github user aalexandrov commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r31113926
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,107 @@
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.math.log
    +import scala.util.hashing.MurmurHash3
    +import scala.collection.mutable.LinkedHashSet;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents.
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val params = transformParameters.get(StopWordParameter)
    +
    +    // Here we will store the words in he form (docId, word, count)
    +    // Count represent the occurrence of "word" in document "docId"
    +    val wordCounts = input
    +      //count the words
    +      .flatMap(t => {
    +      //create tuples docId, word, 1
    +      t._2.map(s => (t._1, s, 1))
    +    })
    +      .filter(t => !transformParameters.apply(StopWordParameter).contains(t._2))
    +      //group by document and word
    +      .groupBy(0, 1)
    +      // calculate the occurrence count of each word in specific document
    +      .sum(2)
    +
    +    val dictionary = wordCounts
    +      .map(t => LinkedHashSet(t._2))
    +      .reduce((set1, set2) => set1 ++ set2)
    +      .map(set => set.zipWithIndex.toMap)
    --- End diff --
    
    Make sure that you have exactly one `map ; flatMap` chain.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by rbraeunlich <gi...@git.apache.org>.
Github user rbraeunlich commented on the pull request:

    https://github.com/apache/flink/pull/730#issuecomment-175473830
  
    Well, someone could. I'm currently out of time to do this. Feel free to apply any changes you think are necessary. Although I still think it's simply a problem with checkstyle.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the pull request:

    https://github.com/apache/flink/pull/730#issuecomment-112824995
  
    Maybe we can also limit the size of the testing data a little bit.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r32623198
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    +          s2 => (t._1, s2.toLowerCase, 1)
    +          )
    +      })
    +    })
    +      .filter(t => !transformParameters.apply(StopWordParameter).contains(t._2))
    +      //group by document and word
    +      .groupBy(0, 1)
    +      // calculate the occurrence count of each word in specific document
    +      .sum(2)
    +
    +    val dictionary = matchedWordCounts
    +      .map(t => LinkedHashSet(t._2))
    +      .reduce((set1, set2) => set1 ++ set2)
    +      //to make sure we only create one dictionary
    +      .setParallelism(1)
    +      .map(set => set.zipWithIndex.toMap)
    +      .flatMap(m => m.toList)
    +
    +    val numberOfWords = matchedWordCounts
    +      .map(t => (t._2))
    +      .distinct(t => t)
    +      .map(t => 1)
    +      .reduce(_ + _);
    +
    +    val idf: DataSet[(String, Double)] = calculateIDF(matchedWordCounts)
    +    val tf: DataSet[(Int, String, Int)] = matchedWordCounts
    +
    +    // docId, word, tfIdf
    +    val tfIdf = tf.join(idf).where(1).equalTo(0) {
    +      (t1, t2) => (t1._1, t1._2, t1._3.toDouble * t2._2)
    +    }
    +
    +    val res = tfIdf.crossWithTiny(numberOfWords)
    +      // docId, word, tfIdf, numberOfWords
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._2))
    +      //assign every word its position
    +      .joinWithTiny(dictionary).where(1).equalTo(0)
    +      //join the tuples (docId, word, tfIdf, numberOfWords, index
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._1._4, t._2._2))
    +      .map(t => {
    +      (t._1, List[(Int, Double)]((t._5, t._3)), t._4)
    +    })
    +      .groupBy(t => t._1)
    +      .reduce((t1, t2) => (t1._1, t1._2 ++ t2._2, t1._3))
    +      .map(t => (t._1, SparseVector.fromCOO(t._3, t._2.toIterable)))
    +
    +    res
    +  }
    +
    +  /**
    +   * Takes the DataSet of DocId, Word and WordCount and calculates the IDF as tuple of word and idf
    +   * @param set
    +   */
    +  private def calculateIDF(set: DataSet[(Int, String, Int)]) = {
    --- End diff --
    
    maybe a more meaningful parameter name could be helpful here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r32623118
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    +          s2 => (t._1, s2.toLowerCase, 1)
    +          )
    +      })
    +    })
    +      .filter(t => !transformParameters.apply(StopWordParameter).contains(t._2))
    +      //group by document and word
    +      .groupBy(0, 1)
    +      // calculate the occurrence count of each word in specific document
    +      .sum(2)
    +
    +    val dictionary = matchedWordCounts
    +      .map(t => LinkedHashSet(t._2))
    +      .reduce((set1, set2) => set1 ++ set2)
    +      //to make sure we only create one dictionary
    +      .setParallelism(1)
    +      .map(set => set.zipWithIndex.toMap)
    +      .flatMap(m => m.toList)
    --- End diff --
    
    Why do you first convert into a map and then into a list?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by FGoessler <gi...@git.apache.org>.
Github user FGoessler commented on the pull request:

    https://github.com/apache/flink/pull/730#issuecomment-112761494
  
    Is something going to happen with this pull request soon, cause we would need a working TF-IDF-Transformer for another project?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by pefi1011 <gi...@git.apache.org>.
Github user pefi1011 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r33427318
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    +          s2 => (t._1, s2.toLowerCase, 1)
    +          )
    +      })
    +    })
    +      .filter(t => !transformParameters.apply(StopWordParameter).contains(t._2))
    +      //group by document and word
    +      .groupBy(0, 1)
    +      // calculate the occurrence count of each word in specific document
    +      .sum(2)
    +
    +    val dictionary = matchedWordCounts
    +      .map(t => LinkedHashSet(t._2))
    +      .reduce((set1, set2) => set1 ++ set2)
    +      //to make sure we only create one dictionary
    +      .setParallelism(1)
    --- End diff --
    
    Done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r32622189
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    --- End diff --
    
    This regex won't capture single character words such as "a" for example.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by pefi1011 <gi...@git.apache.org>.
Github user pefi1011 commented on the pull request:

    https://github.com/apache/flink/pull/730#issuecomment-112822997
  
    Thanks for the feedback..we will improve our code based on it within next 12 days, we are currently all to busy with deadlines from other projects..


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the pull request:

    https://github.com/apache/flink/pull/730#issuecomment-175001442
  
    Can't you use it again?
    
    On Tue, Jan 26, 2016 at 1:48 PM, Ronny Bräunlich <no...@github.com>
    wrote:
    
    > We used scikit-learn as a reference.
    >
    >
    > Am 26.01.2016 um 21:36 schrieb Till Rohrmann <no...@github.com>:
    >
    > > How have you done it in the first place for the poems? Don't you have an
    > > automatic tool for that?
    > >
    > > On Tue, Jan 26, 2016 at 1:21 PM, Ronny Bräunlich <
    > notifications@github.com>
    > > wrote:
    > >
    > > > Somebody has to do the new calculations. I suppose that'll be the
    > biggest
    > > > amount of work.
    > > >
    > > > —
    > > > Reply to this email directly or view it on GitHub
    > > > <https://github.com/apache/flink/pull/730#issuecomment-174979030>.
    >
    > > >
    > > —
    > > Reply to this email directly or view it on GitHub.
    > >
    >
    > —
    > Reply to this email directly or view it on GitHub
    > <https://github.com/apache/flink/pull/730#issuecomment-174997159>.
    >



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by rbraeunlich <gi...@git.apache.org>.
Github user rbraeunlich commented on the pull request:

    https://github.com/apache/flink/pull/730#issuecomment-174957852
  
    Well, one test would have to be changed. That test currently reads the values from ```poems_out.txt```, orders them and compares them afterwards with the calculated numbers (again ordered).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r32619636
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    --- End diff --
    
    This line is longer than 100 characters and thus will be rejected by the scala checkstyle plugin. Have you tried running `mvn clean verify`? This should tell you this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by pefi1011 <gi...@git.apache.org>.
Github user pefi1011 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r33424875
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    +          s2 => (t._1, s2.toLowerCase, 1)
    +          )
    +      })
    +    })
    +      .filter(t => !transformParameters.apply(StopWordParameter).contains(t._2))
    +      //group by document and word
    +      .groupBy(0, 1)
    +      // calculate the occurrence count of each word in specific document
    +      .sum(2)
    +
    +    val dictionary = matchedWordCounts
    +      .map(t => LinkedHashSet(t._2))
    +      .reduce((set1, set2) => set1 ++ set2)
    +      //to make sure we only create one dictionary
    +      .setParallelism(1)
    +      .map(set => set.zipWithIndex.toMap)
    +      .flatMap(m => m.toList)
    +
    +    val numberOfWords = matchedWordCounts
    +      .map(t => (t._2))
    +      .distinct(t => t)
    +      .map(t => 1)
    +      .reduce(_ + _);
    +
    +    val idf: DataSet[(String, Double)] = calculateIDF(matchedWordCounts)
    +    val tf: DataSet[(Int, String, Int)] = matchedWordCounts
    +
    +    // docId, word, tfIdf
    +    val tfIdf = tf.join(idf).where(1).equalTo(0) {
    +      (t1, t2) => (t1._1, t1._2, t1._3.toDouble * t2._2)
    +    }
    +
    +    val res = tfIdf.crossWithTiny(numberOfWords)
    +      // docId, word, tfIdf, numberOfWords
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._2))
    +      //assign every word its position
    +      .joinWithTiny(dictionary).where(1).equalTo(0)
    +      //join the tuples (docId, word, tfIdf, numberOfWords, index
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._1._4, t._2._2))
    +      .map(t => {
    +      (t._1, List[(Int, Double)]((t._5, t._3)), t._4)
    +    })
    +      .groupBy(t => t._1)
    +      .reduce((t1, t2) => (t1._1, t1._2 ++ t2._2, t1._3))
    +      .map(t => (t._1, SparseVector.fromCOO(t._3, t._2.toIterable)))
    +
    +    res
    +  }
    +
    +  /**
    +   * Takes the DataSet of DocId, Word and WordCount and calculates the IDF as tuple of word and idf
    +   * @param set
    +   */
    +  private def calculateIDF(set: DataSet[(Int, String, Int)]) = {
    +    val totalNumberOfDocuments = set
    +      //map the input only to the docId
    +      .map(t => t._1)
    +      .groupBy(i => i)
    +      //reduce to unique docIds
    +      .reduce((t1, t2) => t1)
    --- End diff --
    
    Done



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by pefi1011 <gi...@git.apache.org>.
Github user pefi1011 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r33427182
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    +
    +    val matchedWordCounts = input.flatMap(t => {
    +      t._2.flatMap(s => {
    +        "(?u)\\b\\w\\w+\\b".r findAllIn s map (
    +          s2 => (t._1, s2.toLowerCase, 1)
    +          )
    +      })
    +    })
    +      .filter(t => !transformParameters.apply(StopWordParameter).contains(t._2))
    +      //group by document and word
    +      .groupBy(0, 1)
    +      // calculate the occurrence count of each word in specific document
    +      .sum(2)
    +
    +    val dictionary = matchedWordCounts
    +      .map(t => LinkedHashSet(t._2))
    +      .reduce((set1, set2) => set1 ++ set2)
    +      //to make sure we only create one dictionary
    +      .setParallelism(1)
    +      .map(set => set.zipWithIndex.toMap)
    +      .flatMap(m => m.toList)
    +
    +    val numberOfWords = matchedWordCounts
    +      .map(t => (t._2))
    +      .distinct(t => t)
    +      .map(t => 1)
    +      .reduce(_ + _);
    +
    +    val idf: DataSet[(String, Double)] = calculateIDF(matchedWordCounts)
    +    val tf: DataSet[(Int, String, Int)] = matchedWordCounts
    +
    +    // docId, word, tfIdf
    +    val tfIdf = tf.join(idf).where(1).equalTo(0) {
    +      (t1, t2) => (t1._1, t1._2, t1._3.toDouble * t2._2)
    +    }
    +
    +    val res = tfIdf.crossWithTiny(numberOfWords)
    +      // docId, word, tfIdf, numberOfWords
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._2))
    +      //assign every word its position
    +      .joinWithTiny(dictionary).where(1).equalTo(0)
    +      //join the tuples (docId, word, tfIdf, numberOfWords, index
    +      .map(t => (t._1._1, t._1._2, t._1._3, t._1._4, t._2._2))
    +      .map(t => {
    +      (t._1, List[(Int, Double)]((t._5, t._3)), t._4)
    +    })
    +      .groupBy(t => t._1)
    +      .reduce((t1, t2) => (t1._1, t1._2 ++ t2._2, t1._3))
    +      .map(t => (t._1, SparseVector.fromCOO(t._3, t._2.toIterable)))
    +
    +    res
    +  }
    +
    +  /**
    +   * Takes the DataSet of DocId, Word and WordCount and calculates the IDF as tuple of word and idf
    +   * @param set
    +   */
    +  private def calculateIDF(set: DataSet[(Int, String, Int)]) = {
    +    val totalNumberOfDocuments = set
    +      //map the input only to the docId
    +      .map(t => t._1)
    +      .groupBy(i => i)
    +      //reduce to unique docIds
    +      .reduce((t1, t2) => t1)
    +      .count();
    --- End diff --
    
    Done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [Flink-1999] basic TfidfTransformer

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/730#discussion_r32619656
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/feature/TfIdfTransformer.scala ---
    @@ -0,0 +1,128 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.feature
    +
    +import org.apache.flink.api.scala.{DataSet, _}
    +import org.apache.flink.ml.common.{Parameter, ParameterMap, Transformer}
    +import org.apache.flink.ml.math.SparseVector
    +
    +import scala.collection.mutable.LinkedHashSet
    +import scala.math.log;
    +
    +/**
    + * This transformer calcuates the term-frequency times inverse document frequency for the give DataSet of documents.
    + * The DataSet will be treated as the corpus of documents. The single words will be filtered against the regex:
    + * <code>
    + * (?u)\b\w\w+\b
    + * </code>
    + * <p>
    + * The TF is the frequence of a word inside one document
    + * <p>
    + * The IDF for a word is calculated: log(total number of documents / documents that contain the word) + 1
    + * <p>
    + * This transformer returns a SparseVector where the index is the hash of the word and the value the tf-idf.
    + * @author Ronny Bräunlich
    + * @author Vassil Dimov
    + * @author Filip Perisic
    + */
    +class TfIdfTransformer extends Transformer[(Int, Seq[String]), (Int, SparseVector)] {
    +
    +  override def transform(input: DataSet[(Int /* docId */ , Seq[String] /*The document */ )], transformParameters: ParameterMap): DataSet[(Int, SparseVector)] = {
    --- End diff --
    
    Scaladocs missing here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---