[jira] [Commented] (PARQUET-1989) Deep verification of encrypted files

["Steve Loughran (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PARQUET-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17711367#comment-17711367 ] 

Steve Loughran commented on PARQUET-1989:
-----------------------------------------

you might want to have a design which can do the scan on a spark rdd, where the rdd is simply the deep listFiles(path) scan of the directory tree. This would give the best scale for a massive dataset compared to even some parallelised scan in a single process.

I do have an RDD which can do line-by-line work, with locality of work determined on each file, which lets you schedule the work on the relevant hdfs nodes with the data; unfortunately it needs to be in the o.a.spark package to build
https://github.com/hortonworks-spark/cloud-integration/blob/master/spark-cloud-integration/src/main/scala/org/apache/spark/cloudera/ParallelizedWithLocalityRDD.scala

...that could maybe be added to spark itself.



> Deep verification of encrypted files
> ------------------------------------
>
>                 Key: PARQUET-1989
>                 URL: https://issues.apache.org/jira/browse/PARQUET-1989
>             Project: Parquet
>          Issue Type: New Feature
>          Components: parquet-cli
>            Reporter: Gidon Gershinsky
>            Assignee: Maya Anderson
>            Priority: Major
>             Fix For: 1.14.0
>
>
> A tools that verifies encryption of parquet files in a given folder. Analyzes the footer, and then every module (page headers, pages, column indexes, bloom filters) - making sure they are encrypted (in relevant columns). Potentially checking the encryption keys.
> We'll start with a design doc, open for discussion.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)