[GitHub] [incubator-pinot] fx19880617 opened a new issue #4705: Corrupted segment file generated from pinot-admin segment creation script

[GitBox <gi...@apache.org>]

fx19880617 opened a new issue #4705: Corrupted segment file generated from pinot-admin segment creation script
URL: https://github.com/apache/incubator-pinot/issues/4705
 
 
   Exception happens when server loading segment.
   ```
   2019/10/13 09:13:44.354 ERROR [SingleFileIndexDirectory] [HelixTaskExecutor-message_handle_thread] Missing magic marker in index file: /var/pinot/server/data/index/myTable_OFFLINE/myTable_20190710_0/v3/columns.psf at position: 330711678
   2019/10/13 09:13:44.354 ERROR [SegmentFetcherAndLoader] [HelixTaskExecutor-message_handle_thread] Cannot load segment : myTable_20190710_0 for table myTable_OFFLINE
   java.lang.RuntimeException: Inconsistent data read. Index data file /var/pinot/server/data/index/myTable_OFFLINE/myTable_20190710_0/v3/columns.psf is possibly corrupted
   ```
   
   Sample usage:
   ```
   "/opt/pinot/bin/pinot-admin.sh CreateSegment -format parquet -dataDir /path/to/data/raw/Year1=$year/Month1=$month/Day1=$day -outDir /path/to/data/segment/$year/$month/$day -tableName myTable -segmentName $segmentName -overwrite -schemaFile /path/to/data/schema/schema.json"
   ```
   
   In order to have more insights into the issue, I suggest to add a flag to segment creation script to enable segment data verification if the segment output directory is a remote filesystem.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org