You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/06/21 18:09:17 UTC
[GitHub] [incubator-pinot] SudhaGarimella opened a new issue #5600: Pinot 0.4 Release | Realtime table- Error in creating a segment
SudhaGarimella opened a new issue #5600:
URL: https://github.com/apache/incubator-pinot/issues/5600
This is the Realtime table use case where in data gets streamed from Kafka, during a segment creation - an exception is being thrown. Attached is the exception stack trace from the pinot server. Also attached is the list of file descriptions outputted from lsof command on all processes running on the pinot server.
Same works fine on release 0.3.
[Log-0.4-seg-commit.txt](https://github.com/apache/incubator-pinot/files/4809980/Log-0.4-seg-commit.txt)
[LSOF - listFileDescr-0.4.txt](https://github.com/apache/incubator-pinot/files/4809981/LSOF.-.listFileDescr-0.4.txt)
[tableConfigYML.txt](https://github.com/apache/incubator-pinot/files/4809993/tableConfigYML.txt)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #5600: Pinot 0.4 Release | Realtime table- Error in creating a segment
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #5600:
URL: https://github.com/apache/incubator-pinot/issues/5600#issuecomment-649777045
Is it possible to check the directory permission between two images for the mounted directory inside the container?
You can run `kubectl exec -it pod/pinot-controller-0 -n pino-quickstart bash` to get into the container's bash
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] SudhaGarimella commented on issue #5600: Pinot 0.4 Release | Realtime table- Error in creating a segment
Posted by GitBox <gi...@apache.org>.
SudhaGarimella commented on issue #5600:
URL: https://github.com/apache/incubator-pinot/issues/5600#issuecomment-652022180
This image is post change
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] SudhaGarimella edited a comment on issue #5600: Pinot 0.4 Release | Realtime table- Error in creating a segment
Posted by GitBox <gi...@apache.org>.
SudhaGarimella edited a comment on issue #5600:
URL: https://github.com/apache/incubator-pinot/issues/5600#issuecomment-648783983
@haibow : [seg-create-issue.txt](https://github.com/apache/incubator-pinot/files/4825293/seg-create-issue.txt)
Sorry did not realize that it was truncated. Please check this one. Please check attached file.
Below is extracted from attached file
java.io.IOException: Unable to delete directory /var/pinot/server/data/index/jeepusermap_REALTIME/_tmp/tmp-jeepusermap__0__40__20200623T1858Z-1592999308227/tmp-1592999308462--859419267052440122.
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1541) ~[pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]
at org.apache.commons.io.FileUtils.moveDirectory(FileUtils.java:2756) ~[pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]
at org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl.handlePostCreation(SegmentIndexCreationDriverImpl.java:230) ~[pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]
at org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:199) ~[pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]
at org.apache.pinot.core.realtime.converter.RealtimeSegmentConverter.build(RealtimeSegmentConverter.java:141) ~[pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentInternal(LLRealtimeSegmentDataManager.java:720) [pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentForCommit(LLRealtimeSegmentDataManager.java:671) [pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:588) [pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
2020/06/24 11:49:38.926 INFO [LLRealtimeSegmentDataManager_jeepusermap__1__38__20200623T1849Z] [jeepusermap__1__38__20200623T1849Z] Consumed 19866 events from (rate:331.08344/s), currentOffset=196880089, numRowsConsumedSoFar=99773, numRowsIndexedSoFar=99773
2020/06/24 11:49:39.019 INFO [LLRealtimeSegmentDataManager_jeepusermap__2__54__20200623T1843Z] [jeepusermap__2__54__20200623T1843Z] Consumed 20005 events from (rate:333.39444/s), currentOffset=197074187, numRowsConsumedSoFar=100182, numRowsIndexedSoFar=100182
2020/06/24 11:53:06.296 INFO [LLRealtimeSegmentDataManager_jeepusermap__1__38__20200623T1849Z] [jeepusermap__1__38__20200623T1849Z] Stopping consumption due to row limit nRows=169204 numRowsIndexed=169204, numRowsConsumed=169204
2020/06/24 11:53:06.305 INFO [FileUploadDownloadClient] [jeepusermap__1__38__20200623T1849Z] Sending request: http://pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local:9000/segmentConsumed?name=jeepusermap__1__38__20200623T1849Z&offset=196949520&instance=Server_pinot-server-0.pinot-server-headless.database-prod.svc.cluster.local_8098&reason=rowLimit&rowCount=169204 to controller: pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local, version: Unknown
2020/06/24 11:53:06.306 INFO [ServerSegmentCompletionProtocolHandler] [jeepusermap__1__38__20200623T1849Z] Controller response {"buildTimeSec":-1,"isSplitCommitType":false,"status":"HOLD","offset":196949520} for http://pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local:9000/segmentConsumed?name=jeepusermap__1__38__20200623T1849Z&offset=196949520&instance=Server_pinot-server-0.pinot-server-headless.database-prod.svc.cluster.local_8098&reason=rowLimit&rowCount=169204
2020/06/24 11:53:09.311 INFO [FileUploadDownloadClient] [jeepusermap__1__38__20200623T1849Z] Sending request: http://pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local:9000/segmentConsumed?name=jeepusermap__1__38__20200623T1849Z&offset=196949520&instance=Server_pinot-server-0.pinot-server-headless.database-prod.svc.cluster.local_8098&reason=rowLimit&rowCount=169204 to controller: pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local, version: Unknown
2020/06/24 11:53:09.311 INFO [ServerSegmentCompletionProtocolHandler] [jeepusermap__1__38__20200623T1849Z] Controller response {"buildTimeSec":-1,"isSplitCommitType":false,"status":"HOLD","offset":196949520} for http://pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local:9000/segmentConsumed?name=jeepusermap__1__38__20200623T1849Z&offset=196949520&instance=Server_pinot-server-0.pinot-server-headless.database-prod.svc.cluster.local_8098&reason=rowLimit&rowCount=169204
2020/06/24 11:53:12.315 INFO [FileUploadDownloadClient] [jeepusermap__1__38__20200623T1849Z] Sending request: http://pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local:9000/segmentConsumed?name=jeepusermap__1__38__20200623T1849Z&offset=196949520&instance=Server_pinot-server-0.pinot-server-headless.database-prod.svc.cluster.local_8098&reason=rowLimit&rowCount=169204 to controller: pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local, version: Unknown
2020/06/24 11:53:12.315 INFO [ServerSegmentCompletionProtocolHandler] [jeepusermap__1__38__20200623T1849Z] Controller response {"buildTimeSec":-1,"isSplitCommitType":false,"status":"HOLD","offset":196949520} for http://pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local:9000/segmentConsumed?name=jeepusermap__1__38__20200623T1849Z&offset=196949520&instance=Server_pinot-server-0.pinot-server-headless.database-prod.svc.cluster.local_8098&reason=rowLimit&rowCount=169204
2020/06/24 11:53:15.320 INFO [FileUploadDownloadClient] [jeepusermap__1__38__20200623T1849Z] Sending request: http://pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local:9000/segmentConsumed?name=jeepusermap__1__38__20200623T1849Z&offset=196949520&instance=Server_pinot-server-0.pinot-server-headless.database-prod.svc.cluster.local_8098&reason=rowLimit&rowCount=169204 to controller: pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local, version: Unknown
2020/06/24 11:53:15.320 INFO [ServerSegmentCompletionProtocolHandler] [jeepusermap__1__38__20200623T1849Z] Controller response {"buildTimeSec":-1,"isSplitCommitType":false,"status":"HOLD","offset":196949520} for http://pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local:9000/segmentConsumed?name=jeepusermap__1__38__20200623T1849Z&offset=196949520&instance=Server_pinot-server-0.pinot-server-headless.database-prod.svc.cluster.local_8098&reason=rowLimit&rowCount=169204
Similar logs show that retry is happening. Since all consuming threads on all servers threw the same issue, no segment got created in controller also and all 3 consuming threads are down.
And from azure portal I am able to delete the directory mentioned in above exception without any issues
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] sudha-garimella commented on issue #5600: Pinot 0.4 Release | Realtime table- Error in creating a segment
Posted by GitBox <gi...@apache.org>.
sudha-garimella commented on issue #5600:
URL: https://github.com/apache/incubator-pinot/issues/5600#issuecomment-651223914
I checked the folder permissions for 0.4 they have read write and execute. But i cant check for previous release since those folders are already deleted.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on issue #5600: Pinot 0.4 Release | Realtime table- Error in creating a segment
Posted by GitBox <gi...@apache.org>.
kishoreg commented on issue #5600:
URL: https://github.com/apache/incubator-pinot/issues/5600#issuecomment-649658904
@fx19880617 any changes to the way we start k8s containers?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] SudhaGarimella edited a comment on issue #5600: Pinot 0.4 Release | Realtime table- Error in creating a segment
Posted by GitBox <gi...@apache.org>.
SudhaGarimella edited a comment on issue #5600:
URL: https://github.com/apache/incubator-pinot/issues/5600#issuecomment-648783983
[seg-create-issue.txt](https://github.com/apache/incubator-pinot/files/4825293/seg-create-issue.txt)
Sorry did not realize that it was truncated. Please check this one. Please check attached file.
Below is extracted from attached file
java.io.IOException: Unable to delete directory /var/pinot/server/data/index/jeepusermap_REALTIME/_tmp/tmp-jeepusermap__0__40__20200623T1858Z-1592999308227/tmp-1592999308462--859419267052440122.
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1541) ~[pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]
at org.apache.commons.io.FileUtils.moveDirectory(FileUtils.java:2756) ~[pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]
at org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl.handlePostCreation(SegmentIndexCreationDriverImpl.java:230) ~[pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]
at org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:199) ~[pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]
at org.apache.pinot.core.realtime.converter.RealtimeSegmentConverter.build(RealtimeSegmentConverter.java:141) ~[pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentInternal(LLRealtimeSegmentDataManager.java:720) [pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentForCommit(LLRealtimeSegmentDataManager.java:671) [pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:588) [pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
2020/06/24 11:49:38.926 INFO [LLRealtimeSegmentDataManager_jeepusermap__1__38__20200623T1849Z] [jeepusermap__1__38__20200623T1849Z] Consumed 19866 events from (rate:331.08344/s), currentOffset=196880089, numRowsConsumedSoFar=99773, numRowsIndexedSoFar=99773
2020/06/24 11:49:39.019 INFO [LLRealtimeSegmentDataManager_jeepusermap__2__54__20200623T1843Z] [jeepusermap__2__54__20200623T1843Z] Consumed 20005 events from (rate:333.39444/s), currentOffset=197074187, numRowsConsumedSoFar=100182, numRowsIndexedSoFar=100182
2020/06/24 11:53:06.296 INFO [LLRealtimeSegmentDataManager_jeepusermap__1__38__20200623T1849Z] [jeepusermap__1__38__20200623T1849Z] Stopping consumption due to row limit nRows=169204 numRowsIndexed=169204, numRowsConsumed=169204
2020/06/24 11:53:06.305 INFO [FileUploadDownloadClient] [jeepusermap__1__38__20200623T1849Z] Sending request: http://pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local:9000/segmentConsumed?name=jeepusermap__1__38__20200623T1849Z&offset=196949520&instance=Server_pinot-server-0.pinot-server-headless.database-prod.svc.cluster.local_8098&reason=rowLimit&rowCount=169204 to controller: pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local, version: Unknown
2020/06/24 11:53:06.306 INFO [ServerSegmentCompletionProtocolHandler] [jeepusermap__1__38__20200623T1849Z] Controller response {"buildTimeSec":-1,"isSplitCommitType":false,"status":"HOLD","offset":196949520} for http://pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local:9000/segmentConsumed?name=jeepusermap__1__38__20200623T1849Z&offset=196949520&instance=Server_pinot-server-0.pinot-server-headless.database-prod.svc.cluster.local_8098&reason=rowLimit&rowCount=169204
2020/06/24 11:53:09.311 INFO [FileUploadDownloadClient] [jeepusermap__1__38__20200623T1849Z] Sending request: http://pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local:9000/segmentConsumed?name=jeepusermap__1__38__20200623T1849Z&offset=196949520&instance=Server_pinot-server-0.pinot-server-headless.database-prod.svc.cluster.local_8098&reason=rowLimit&rowCount=169204 to controller: pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local, version: Unknown
2020/06/24 11:53:09.311 INFO [ServerSegmentCompletionProtocolHandler] [jeepusermap__1__38__20200623T1849Z] Controller response {"buildTimeSec":-1,"isSplitCommitType":false,"status":"HOLD","offset":196949520} for http://pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local:9000/segmentConsumed?name=jeepusermap__1__38__20200623T1849Z&offset=196949520&instance=Server_pinot-server-0.pinot-server-headless.database-prod.svc.cluster.local_8098&reason=rowLimit&rowCount=169204
2020/06/24 11:53:12.315 INFO [FileUploadDownloadClient] [jeepusermap__1__38__20200623T1849Z] Sending request: http://pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local:9000/segmentConsumed?name=jeepusermap__1__38__20200623T1849Z&offset=196949520&instance=Server_pinot-server-0.pinot-server-headless.database-prod.svc.cluster.local_8098&reason=rowLimit&rowCount=169204 to controller: pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local, version: Unknown
2020/06/24 11:53:12.315 INFO [ServerSegmentCompletionProtocolHandler] [jeepusermap__1__38__20200623T1849Z] Controller response {"buildTimeSec":-1,"isSplitCommitType":false,"status":"HOLD","offset":196949520} for http://pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local:9000/segmentConsumed?name=jeepusermap__1__38__20200623T1849Z&offset=196949520&instance=Server_pinot-server-0.pinot-server-headless.database-prod.svc.cluster.local_8098&reason=rowLimit&rowCount=169204
2020/06/24 11:53:15.320 INFO [FileUploadDownloadClient] [jeepusermap__1__38__20200623T1849Z] Sending request: http://pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local:9000/segmentConsumed?name=jeepusermap__1__38__20200623T1849Z&offset=196949520&instance=Server_pinot-server-0.pinot-server-headless.database-prod.svc.cluster.local_8098&reason=rowLimit&rowCount=169204 to controller: pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local, version: Unknown
2020/06/24 11:53:15.320 INFO [ServerSegmentCompletionProtocolHandler] [jeepusermap__1__38__20200623T1849Z] Controller response {"buildTimeSec":-1,"isSplitCommitType":false,"status":"HOLD","offset":196949520} for http://pinot-controller-1.pinot-controller-headless.database-prod.svc.cluster.local:9000/segmentConsumed?name=jeepusermap__1__38__20200623T1849Z&offset=196949520&instance=Server_pinot-server-0.pinot-server-headless.database-prod.svc.cluster.local_8098&reason=rowLimit&rowCount=169204
Similar logs show that retry is happening. Since all consuming threads on all servers threw the same issue, no segment got created in controller also and all 3 consuming threads are down.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] SudhaGarimella commented on issue #5600: Pinot 0.4 Release | Realtime table- Error in creating a segment
Posted by GitBox <gi...@apache.org>.
SudhaGarimella commented on issue #5600:
URL: https://github.com/apache/incubator-pinot/issues/5600#issuecomment-648783983
[seg-create-issue.txt](https://github.com/apache/incubator-pinot/files/4825293/seg-create-issue.txt)
Sorry did not realize that it was truncated. Please check this one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] SudhaGarimella commented on issue #5600: Pinot 0.4 Release | Realtime table- Error in creating a segment
Posted by GitBox <gi...@apache.org>.
SudhaGarimella commented on issue #5600:
URL: https://github.com/apache/incubator-pinot/issues/5600#issuecomment-649028117
@mcvsubbu : so 2 things
1. the azure portal path is mounted into the server, so isnt that like a local file for pinot server?
2. what we did today was we first had used 0.3 release, then did a rolling update (changed the img to 0.4). When using 0.3 you can see we created about 39 segments, when upgraded to 0.4, the first segment it tried to create had failed. So the point I am saying is - same setup worked well with 0.3 release, when v did a rolling update to 0.4, seg creation failed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] SudhaGarimella commented on issue #5600: Pinot 0.4 Release | Realtime table- Error in creating a segment
Posted by GitBox <gi...@apache.org>.
SudhaGarimella commented on issue #5600:
URL: https://github.com/apache/incubator-pinot/issues/5600#issuecomment-652017600
As suggested by @fx19880617 , I updated the storage class for server to default (previous value was azurefile) and then segments are getting created now without any exception.
As per recommended approach, storage class for server has to be azuredisk which is the default and not azurefile.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on issue #5600: Pinot 0.4 Release | Realtime table- Error in creating a segment
Posted by GitBox <gi...@apache.org>.
kishoreg commented on issue #5600:
URL: https://github.com/apache/incubator-pinot/issues/5600#issuecomment-647168833
There was no change in the code related to segment creation.
@mcvsubbu @npawar @Jackie-Jiang any idea whats going on here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on issue #5600: Pinot 0.4 Release | Realtime table- Error in creating a segment
Posted by GitBox <gi...@apache.org>.
kishoreg commented on issue #5600:
URL: https://github.com/apache/incubator-pinot/issues/5600#issuecomment-650499099
@SudhaGarimella can you provide more info
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] haibow commented on issue #5600: Pinot 0.4 Release | Realtime table- Error in creating a segment
Posted by GitBox <gi...@apache.org>.
haibow commented on issue #5600:
URL: https://github.com/apache/incubator-pinot/issues/5600#issuecomment-647259914
The last few lines of the log shows
```
2020/06/21 11:26:45.012 INFO [SegmentDictionaryCreator] [jeepusermap__1__0__20200621T1125Z] Using fixed bytes value dictionary for column: field4, size: 0
2020/06/21 11:26:45.012 INFO [SegmentDictionaryCreator] [jeepusermap__1__0__20200621T1125Z] Created dictionary for STRING column: field4 with cardinality: 1, max length in bytes: 0, range: to
2020/06/21 11:26:45.036 INFO [SegmentIndexCreationDriverImpl] [jeepusermap__2__0__20200621T1125Z] Start building IndexCreator!
2020/06/21 11:26:45.102 INFO [SegmentIndexCreationDriverImpl] [jeepusermap__0__0__20200621T1125Z] Start building IndexCreator!
2020/06/21 11:26:45.113 INFO [SegmentIndexCreationDriverImpl] [jeepusermap__1__0__20200621T1125Z] Start building IndexCreator!
2020/06/21 11:26:45.433 INFO [SegmentIndexCreationDriverImpl] [jeepusermap__2__0__20200621T1125Z] Finished records indexing in IndexCreator!
2020/06/21 11:26:45.470 INFO [SegmentIndexCreationDriverImpl] [jeepusermap__0__0__20200621T1125Z] Finished records indexing in IndexCreator!
2020/06/21 11:26:45.473 INFO [SegmentIndexCreationDriverImpl] [jeepusermap__1__0__20200621T1125Z] Finished records indexing in IndexCreator!
2020/06/21 11:26:48.632 INFO [SegmentIndexCreationDriverImpl] [jeepusermap__2__0__20200621T1125Z] Finished segment seal!
2020/06/21 11:26:48.638 INFO [SegmentIndexCreationDriverImpl] [jeepusermap__1__0__20200621T1125Z] Finished segment seal!
2020/06/21 11:26:48.674 INFO [SegmentIndexCreationDriverImpl] [jeepusermap__0__0__20200621T1125Z] Finished segment seal!
2020/06/21 11:27:22.204 ERROR [LLRealtimeSegmentDataManager_jeepusermap__0__0__20200621T1125Z] [jeepusermap__0__0__20200621T1125Z] Could not build segment
java.io.IOException: Unable to delete directory /var/pinot/server/data/index/jeepusermap_REALTIME/_tmp/tmp-jeepusermap__0__0__20200621T1125Z-1592738788709/tmp-1592738788896-2704508350342974686.
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1541) ~[pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]
at org.apache.commons.io.FileUtils.move
```
Looks like the segment sealed successfully but failed to delete the tmp folder.
If you could paste more logs (esp after the exception) we might have more clues. Might be some permission issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on issue #5600: Pinot 0.4 Release | Realtime table- Error in creating a segment
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #5600:
URL: https://github.com/apache/incubator-pinot/issues/5600#issuecomment-648888000
@SudhaGarimella I am not sure how azure permissions work. Things to examine would be:
(1) What user did you run as when deleting the file from azure portal?
(2) what user does the pinot server executable run as?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 edited a comment on issue #5600: Pinot 0.4 Release | Realtime table- Error in creating a segment
Posted by GitBox <gi...@apache.org>.
fx19880617 edited a comment on issue #5600:
URL: https://github.com/apache/incubator-pinot/issues/5600#issuecomment-649777045
Is it possible to check the directory permission between two images for the mounted directory inside the container?
You can run `kubectl exec -it pod/pinot-controller-0 -n pinot-quickstart bash` to get into the container's bash
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org