You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "dang-stripe (via GitHub)" <gi...@apache.org> on 2023/10/12 21:05:07 UTC
[PR] pinot-s3: add woodstox-core to dependencies and fix stack trace [pinot]
dang-stripe opened a new pull request, #11799:
URL: https://github.com/apache/pinot/pull/11799
cc @Jackie-Jiang @xiangfu0
We ran into an issue where our custom plugin jars were deleted from disk and our realtime tables started failing to upload their segments. This occurred without restarting the process.
It turns out the AWS SDK used by pinot-s3 has a transitive dependency on woodstox-core. Our custom plugin that was deleted also has a transitive dep on woodstox-core so it was providing it. This PR makes the dependency explicit.
For context, our classpath looks something like this for the realtime servers:
```
/pinot/lib/*
/pinot/plugins/pinot-file-system/pinot-s3/pinot-s3-0.13.0-2023-10-10-570198069b-SNAPSHOT-shaded.jar
/pinot/plugins/pinot-stream-ingestion/pinot-kafka-2.0/pinot-kafka-2.0-0.13.0-2023-10-10-570198069b-SNAPSHOT-shaded.jar
/pinot/plugins/pinot-stream-ingestion/stripe-ingestion/stripe-ingestion-0.13.0-2023-10-10-570198069b-SNAPSHOT-shaded.jar
# our custom jar
/pinot/custom-plugins/pinot-plugin-bson-decoder/pinot-plugin-bson-decoder-0.0.1.jar
```
Here's the full stacktrace. You can see it no longer can find the pinot-plugin-bson-decoder jar.
```
[2023-10-10 04:21:34.997451] WARN [PinotFSSegmentUploader] [ExampleSegment_1234:17] Failed to upload file /pinot/data/index/ExampleTable/ExampleSegment_1234.tar.gz of segment ExampleSegment_1234 for table ExampleTable
[2023-10-10 04:21:34.997498] java.util.concurrent.ExecutionException: javax.xml.stream.FactoryConfigurationError: Provider for class javax.xml.stream.XMLInputFactory cannot be created
[2023-10-10 04:21:34.997509] at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]
[2023-10-10 04:21:34.997525] at java.util.concurrent.FutureTask.get(FutureTask.java:205) ~[?:?]
[2023-10-10 04:21:34.997544] at org.apache.pinot.core.data.manager.realtime.PinotFSSegmentUploader.uploadSegment(PinotFSSegmentUploader.java:98) ~[pinot-all-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-jar-with-dependencies.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
...
[2023-10-10 04:21:34.997681] at java.lang.Thread.run(Thread.java:829) [?:?]
[2023-10-10 04:21:34.997691] Caused by: javax.xml.stream.FactoryConfigurationError: Provider for class javax.xml.stream.XMLInputFactory cannot be created
[2023-10-10 04:21:34.997700] at javax.xml.stream.FactoryFinder.findServiceProvider(FactoryFinder.java:366) ~[?:?]
...
[2023-10-10 04:21:34.998516] Caused by: java.lang.RuntimeException: Provider for class javax.xml.stream.XMLInputFactory cannot be created
[2023-10-10 04:21:34.998525] at javax.xml.stream.FactoryFinder.findServiceProvider(FactoryFinder.java:363) ~[?:?]
...
[2023-10-10 04:21:34.999350] Caused by: java.util.ServiceConfigurationError: javax.xml.stream.XMLInputFactory: Error accessing configuration file
[2023-10-10 04:21:34.999362] at java.util.ServiceLoader.fail(ServiceLoader.java:582) ~[?:?]
[2023-10-10 04:21:34.999370] at java.util.ServiceLoader$LazyClassPathLookupIterator.parse(ServiceLoader.java:1173) ~[?:?]
[2023-10-10 04:21:34.999378] at java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(ServiceLoader.java:1206) ~[?:?]
[2023-10-10 04:21:34.999390] at java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1221) ~[?:?]
[2023-10-10 04:21:34.999397] at java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1265) ~[?:?]
[2023-10-10 04:21:34.999407] at java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1300) ~[?:?]
[2023-10-10 04:21:34.999417] at java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1385) ~[?:?]
[2023-10-10 04:21:34.999424] at javax.xml.stream.FactoryFinder$1.run(FactoryFinder.java:348) ~[?:?]
[2023-10-10 04:21:34.999431] at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
[2023-10-10 04:21:34.999439] at javax.xml.stream.FactoryFinder.findServiceProvider(FactoryFinder.java:337) ~[?:?]
[2023-10-10 04:21:34.999446] at javax.xml.stream.FactoryFinder.find(FactoryFinder.java:309) ~[?:?]
[2023-10-10 04:21:34.999454] at javax.xml.stream.FactoryFinder.find(FactoryFinder.java:222) ~[?:?]
[2023-10-10 04:21:34.999461] at javax.xml.stream.XMLInputFactory.newInstance(XMLInputFactory.java:161) ~[?:?]
[2023-10-10 04:21:34.999475] at software.amazon.awssdk.protocols.query.unmarshall.XmlDomParser.createXmlInputFactory(XmlDomParser.java:124) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999484] at java.lang.ThreadLocal$SuppliedThreadLocal.initialValue(ThreadLocal.java:305) ~[?:?]
[2023-10-10 04:21:34.999494] at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:195) ~[?:?]
[2023-10-10 04:21:34.999504] at java.lang.ThreadLocal.get(ThreadLocal.java:172) ~[?:?]
[2023-10-10 04:21:34.999518] at software.amazon.awssdk.protocols.query.unmarshall.XmlDomParser.parse(XmlDomParser.java:44) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999533] at software.amazon.awssdk.protocols.xml.internal.unmarshall.XmlResponseParserUtils.parse(XmlResponseParserUtils.java:50) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999549] at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.parseResponse(AwsXmlPredicatedResponseHandler.java:114) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999564] at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:95) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999583] at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:84) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999601] at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:42) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999625] at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:94) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999649] at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$5(BaseClientHandler.java:229) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999668] at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999683] at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999699] at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999714] at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999729] at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999744] at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999758] at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999774] at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999790] at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999804] at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999819] at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999835] at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999849] at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999866] at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999882] at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999897] at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999912] at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999927] at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999942] at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999957] at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999972] at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:34.999989] at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:35.000005] at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:35.000020] at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:35.000037] at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:128) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:35.000051] at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:154) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:35.000068] at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:107) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:35.000083] at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:162) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:35.000096] at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:91) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:35.000128] at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:35.000142] at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:35.000157] at software.amazon.awssdk.services.s3.DefaultS3Client.listObjectsV2(DefaultS3Client.java:5614) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:35.000170] at org.apache.pinot.plugin.filesystem.S3PinotFS.isDirectory(S3PinotFS.java:653) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:35.000182] at org.apache.pinot.plugin.filesystem.S3PinotFS.exists(S3PinotFS.java:457) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:35.000196] at org.apache.pinot.spi.filesystem.NoClosePinotFS.exists(NoClosePinotFS.java:74) ~[pinot-all-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-jar-with-dependencies.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:35.000213] at org.apache.pinot.core.data.manager.realtime.PinotFSSegmentUploader.lambda$uploadSegment$0(PinotFSSegmentUploader.java:82) ~[pinot-all-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-jar-with-dependencies.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
[2023-10-10 04:21:35.000220] at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
[2023-10-10 04:21:35.000228] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
[2023-10-10 04:21:35.000237] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
[2023-10-10 04:21:35.000241] ... 1 more
[2023-10-10 04:21:35.000253] Caused by: java.nio.file.NoSuchFileException: /pinot/plugins/pinot-plugin-bson-decoder/pinot-plugin-bson-decoder-0.0.1.jar
[2023-10-10 04:21:35.000261] at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
[2023-10-10 04:21:35.000269] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
[2023-10-10 04:21:35.000276] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
[2023-10-10 04:21:35.000285] at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) ~[?:?]
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] pinot-s3: add woodstox-core to dependencies and fix stack trace [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #11799:
URL: https://github.com/apache/pinot/pull/11799#issuecomment-1760412460
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11799?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#11799](https://app.codecov.io/gh/apache/pinot/pull/11799?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (a794637) into [master](https://app.codecov.io/gh/apache/pinot/commit/17ffa3e305b0b4da652ba3915ec9c2db65d66b06?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (17ffa3e) will **decrease** coverage by `48.69%`.
> Report is 10 commits behind head on master.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #11799 +/- ##
=============================================
- Coverage 63.09% 14.41% -48.69%
+ Complexity 1117 201 -916
=============================================
Files 2342 2343 +1
Lines 125920 126312 +392
Branches 19370 19419 +49
=============================================
- Hits 79454 18208 -61246
- Misses 40812 106559 +65747
+ Partials 5654 1545 -4109
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/11799/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Ξ | |
|---|---|---|
| [integration](https://app.codecov.io/gh/apache/pinot/pull/11799/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/11799/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/11799/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/11799/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.41% <0.00%> (-48.64%)` | :arrow_down: |
| [java-17](https://app.codecov.io/gh/apache/pinot/pull/11799/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-20](https://app.codecov.io/gh/apache/pinot/pull/11799/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/11799/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.41% <0.00%> (-48.69%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/11799/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.41% <0.00%> (-48.68%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/11799/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/11799/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.41% <0.00%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files](https://app.codecov.io/gh/apache/pinot/pull/11799?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Ξ | |
|---|---|---|
| [.../data/manager/realtime/PinotFSSegmentUploader.java](https://app.codecov.io/gh/apache/pinot/pull/11799?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUGlub3RGU1NlZ21lbnRVcGxvYWRlci5qYXZh) | `0.00% <0.00%> (-75.56%)` | :arrow_down: |
... and [1520 files with indirect coverage changes](https://app.codecov.io/gh/apache/pinot/pull/11799/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: Weβre building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] pinot-s3: add woodstox-core to dependencies and fix stack trace [pinot]
Posted by "dang-stripe (via GitHub)" <gi...@apache.org>.
dang-stripe commented on code in PR #11799:
URL: https://github.com/apache/pinot/pull/11799#discussion_r1364448365
##########
pinot-plugins/pinot-file-system/pinot-s3/pom.xml:
##########
@@ -39,6 +39,7 @@
<http.core.version>4.4.13</http.core.version>
<s3mock.version>2.12.2</s3mock.version>
<javax.version>3.1.0</javax.version>
+ <woodstox.version>5.3.0</woodstox.version>
Review Comment:
@abhioncbr @Jackie-Jiang i feel like it'd be better to keep the dependency closer to the plugins that actually need it? for this case, this dep is specifically used by the aws sdk.
```
[2023-10-10 04:21:35.000157] at software.amazon.awssdk.services.s3.DefaultS3Client.listObjectsV2(DefaultS3Client.java:5614) ~[pinot-s3-0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-shaded.jar:0.13.0-2023-10-09-3ed8e67fe7-SNAPSHOT-3ed8e67fe789f4f16c78c2d9fcad6a6bf27f0a79]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] pinot-s3: add woodstox-core to dependencies and fix stack trace [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #11799:
URL: https://github.com/apache/pinot/pull/11799#discussion_r1366119485
##########
pinot-plugins/pinot-file-system/pinot-s3/pom.xml:
##########
@@ -39,6 +39,7 @@
<http.core.version>4.4.13</http.core.version>
<s3mock.version>2.12.2</s3mock.version>
<javax.version>3.1.0</javax.version>
+ <woodstox.version>5.3.0</woodstox.version>
Review Comment:
@dang-stripe Downgrading a dependency is usually risky. Can you check if changing the version in root to `6.4.0` works? Usually libraries are backward compatible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] pinot-s3: add woodstox-core to dependencies and fix stack trace [pinot]
Posted by "dang-stripe (via GitHub)" <gi...@apache.org>.
dang-stripe commented on code in PR #11799:
URL: https://github.com/apache/pinot/pull/11799#discussion_r1365944790
##########
pinot-plugins/pinot-file-system/pinot-s3/pom.xml:
##########
@@ -39,6 +39,7 @@
<http.core.version>4.4.13</http.core.version>
<s3mock.version>2.12.2</s3mock.version>
<javax.version>3.1.0</javax.version>
+ <woodstox.version>5.3.0</woodstox.version>
Review Comment:
@Jackie-Jiang @abhioncbr i added a `dependencyManagement` section to make the pinot-file-system plugins use the same version of woodstox, but ran into a conflict in CI:
```
Error: Dependency convergence error for com.fasterxml.woodstox:woodstox-core:jar:5.3.0 paths to dependency are:
Error: +-org.apache.pinot:pinot-tools:jar:1.1.0-SNAPSHOT
Error: +-org.apache.hadoop:hadoop-common:jar:3.2.4:compile
Error: +-com.fasterxml.woodstox:woodstox-core:jar:5.3.0:compile
Error: and
Error: +-org.apache.pinot:pinot-tools:jar:1.1.0-SNAPSHOT
Error: +-org.apache.pinot:pinot-s3:jar:1.1.0-SNAPSHOT:runtime
Error: +-com.fasterxml.woodstox:woodstox-core:jar:6.4.0:runtime
```
i think 5.3.0 is the version required by [hadoop-common 3.2.4](https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common/3.2.4) at the root level of the pinot project. it seems like the option here are to move the `dependencyManagement` version clause to the root level and set as 5.3.0 which downgrades the version used by `pinot-adls`. does that seem reasonable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] pinot-s3: add woodstox-core to dependencies and fix stack trace [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #11799:
URL: https://github.com/apache/pinot/pull/11799#discussion_r1364648035
##########
pinot-plugins/pinot-file-system/pinot-s3/pom.xml:
##########
@@ -39,6 +39,7 @@
<http.core.version>4.4.13</http.core.version>
<s3mock.version>2.12.2</s3mock.version>
<javax.version>3.1.0</javax.version>
+ <woodstox.version>5.3.0</woodstox.version>
Review Comment:
The dependencies should be added here, but the version is better managed at root level to avoid unnecessary version conflict. I don't know why we have some versions managed here. @KKcorps any specific reason why we have different version here vs root for javax?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] pinot-s3: add woodstox-core to dependencies and fix stack trace [pinot]
Posted by "dang-stripe (via GitHub)" <gi...@apache.org>.
dang-stripe commented on code in PR #11799:
URL: https://github.com/apache/pinot/pull/11799#discussion_r1366190037
##########
pinot-plugins/pinot-file-system/pinot-s3/pom.xml:
##########
@@ -39,6 +39,7 @@
<http.core.version>4.4.13</http.core.version>
<s3mock.version>2.12.2</s3mock.version>
<javax.version>3.1.0</javax.version>
+ <woodstox.version>5.3.0</woodstox.version>
Review Comment:
π i bumped it to 6.4.0. the release docs say 6.x are all backward compatible w/ 5.x: https://github.com/FasterXML/woodstox/wiki/Woodstox-Release-6.0
> Woodstox 6.0 is the first Java 9 module info compatible version: it is, however, fully API-compatible 5.x and does not require newer JDK version either.
i can also re-test this again on one of our clusters.
##########
pinot-plugins/pinot-file-system/pinot-s3/pom.xml:
##########
@@ -39,6 +39,7 @@
<http.core.version>4.4.13</http.core.version>
<s3mock.version>2.12.2</s3mock.version>
<javax.version>3.1.0</javax.version>
+ <woodstox.version>5.3.0</woodstox.version>
Review Comment:
π i bumped it to 6.4.0. the release docs say 6.x are all backward compatible w/ 5.x: https://github.com/FasterXML/woodstox/wiki/Woodstox-Release-6.0
> Woodstox 6.0 is the first Java 9 module info compatible version: it is, however, fully API-compatible 5.x and does not require newer JDK version either.
i can also re-test this again on one of our clusters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] pinot-s3: add woodstox-core to dependencies and fix stack trace [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #11799:
URL: https://github.com/apache/pinot/pull/11799
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] pinot-s3: add woodstox-core to dependencies and fix stack trace [pinot]
Posted by "abhioncbr (via GitHub)" <gi...@apache.org>.
abhioncbr commented on code in PR #11799:
URL: https://github.com/apache/pinot/pull/11799#discussion_r1357586948
##########
pinot-plugins/pinot-file-system/pinot-s3/pom.xml:
##########
@@ -39,6 +39,7 @@
<http.core.version>4.4.13</http.core.version>
<s3mock.version>2.12.2</s3mock.version>
<javax.version>3.1.0</javax.version>
+ <woodstox.version>5.3.0</woodstox.version>
Review Comment:
We have the woodstox-core dependency in [pinot-adls ](https://github.com/apache/pinot/blob/master/pinot-plugins/pinot-file-system/pinot-adls/pom.xml#L78) as well.
Does it make sense to shift this common dependency at the [pinot-file-system ](https://github.com/apache/pinot/blob/master/pinot-plugins/pinot-file-system/pom.xml)level?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org