You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemds.apache.org by "esracosgun (via GitHub)" <gi...@apache.org> on 2025/07/14 18:14:35 UTC
[PR] AMLS Exercise: Builtin for tuples deduplication [systemds]
esracosgun opened a new pull request, #2293:
URL: https://github.com/apache/systemds/pull/2293
This pull request implements a deduplication pipeline for string tuples using distributed representations and LSH, as required for the assignment. The following steps are covered:
- Distributed Representation: Each tuple is mapped to a dense vector using average GloVe embeddings.
- LSH-based Blocking: Similar tuples are grouped into candidate buckets using random hyperplanes (LSH).
- Similarity Computation: Candidate pairs are compared using a similarity measure (cosine or euclidean)
- Duplicate Filtering: Tuples exceeding a similarity threshold are marked as duplicates; only the first occurrence is retained as unique.
Outputs:
- Y_unique: Deduplicated tuples (first occurrence only).
- Y_duplicates: All detected duplicates removed from the input.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SYSTEMDS-3178] Builtin for tuples deduplication [systemds]
Posted by "mboehm7 (via GitHub)" <gi...@apache.org>.
mboehm7 commented on PR #2293:
URL: https://github.com/apache/systemds/pull/2293#issuecomment-3232505191
LGTM - thanks for the contribution @esracosgun @zohrehasadi00. During the merge I resolved remaining warnings, and put a fixme on the result comparison in the test. In another project, we should then vectorize the functions of this new builtin function (and there is potential for this vectorization). Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SYSTEMDS-3178] Builtin for tuples deduplication [systemds]
Posted by "mboehm7 (via GitHub)" <gi...@apache.org>.
mboehm7 closed pull request #2293: [SYSTEMDS-3178] Builtin for tuples deduplication
URL: https://github.com/apache/systemds/pull/2293
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org