You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pinot.apache.org by Pinot Slack Email Digest <sn...@apache.org> on 2020/12/24 02:00:13 UTC
Apache Pinot Daily Email Digest (2020-12-23)
### _#general_
**@mo.aladham:** @mo.aladham has joined the channel
### _#random_
**@mo.aladham:** @mo.aladham has joined the channel
### _#troubleshooting_
**@mo.aladham:** @mo.aladham has joined the channel
**@laxman:** Hi, anyone facing issues with Segment purging with GCS as deep
store.
**@laxman:** We are facing the following issue ```2020/12/22 02:35:25.329
ERROR [SegmentDeletionManager] [pool-6-thread-1] Had trouble deleting
directories: java.io.IOException: java.io.IOException:
java.net.URISyntaxException: Expected scheme-specific part at index 5: file:
at
org.apache.pinot.plugin.filesystem.GcsPinotFS.listFiles(GcsPinotFS.java:322)
~[pinot-gcs-0.6.0-shaded.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at
org.apache.pinot.controller.helix.core.SegmentDeletionManager.removeAgedDeletedSegments(SegmentDeletionManager.java:233)
~[pinot-all-0.6.0-jar-with-
dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at
org.apache.pinot.controller.helix.core.retention.RetentionManager.postprocess(RetentionManager.java:89)
~[pinot-all-0.6.0-jar-with-
dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at
org.apache.pinot.controller.helix.core.periodictask.ControllerPeriodicTask.postprocess(ControllerPeriodicTask.java:131)
~[pinot-all-0.6.0-jar-with-
dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at
org.apache.pinot.controller.helix.core.periodictask.ControllerPeriodicTask.processTables(ControllerPeriodicTask.java:97)
~[pinot-all-0.6.0-jar-with-
dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at
org.apache.pinot.controller.helix.core.periodictask.ControllerPeriodicTask.runTask(ControllerPeriodicTask.java:68)
~[pinot-all-0.6.0-jar-with-
dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at
org.apache.pinot.core.periodictask.BasePeriodicTask.run(BasePeriodicTask.java:120)
~[pinot-all-0.6.0-jar-with-
dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at
org.apache.pinot.core.periodictask.PeriodicTaskScheduler.lambda$start$0(PeriodicTaskScheduler.java:73)
~[pinot-all-0.6.0-jar-with-
dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?] at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
[?:?] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by:
java.io.IOException: java.net.URISyntaxException: Expected scheme-specific
part at index 5: file: at
org.apache.pinot.plugin.filesystem.GcsPinotFS.getBase(GcsPinotFS.java:135)
~[pinot-gcs-0.6.0-shaded.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at
org.apache.pinot.plugin.filesystem.GcsPinotFS.normalizeToDirectoryPrefix(GcsPinotFS.java:105)
~[pinot-gcs-0.6.0-shaded.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at
org.apache.pinot.plugin.filesystem.GcsPinotFS.listFiles(GcsPinotFS.java:307)
~[pinot-gcs-0.6.0-shaded.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
... 13 more Caused by: java.net.URISyntaxException: Expected scheme-specific
part at index 5: file: at java.net.URI$Parser.fail(URI.java:2913) ~[?:?] at
java.net.URI$Parser.failExpecting(URI.java:2919) ~[?:?] at
java.net.URI$Parser.parse(URI.java:3119) ~[?:?] at
java.net.URI.<init>(URI.java:685) ~[?:?] at java.net.URI.<init>(URI.java:786)
~[?:?] at
org.apache.pinot.plugin.filesystem.GcsPinotFS.getBase(GcsPinotFS.java:133)
~[pinot-gcs-0.6.0-shaded.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at
org.apache.pinot.plugin.filesystem.GcsPinotFS.normalizeToDirectoryPrefix(GcsPinotFS.java:105)
~[pinot-gcs-0.6.0-shaded.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at
org.apache.pinot.plugin.filesystem.GcsPinotFS.listFiles(GcsPinotFS.java:307)
~[pinot-gcs-0.6.0-shaded.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
... 13 more```
**@laxman:** Had gone through this codepath multiple times but I couldn’t see
any issue with config or code.
**@dlavoie:** The exceptions complains of a malformed URI ```
java.net.URISyntaxException: Expected scheme-specific part at index 5:
file:``` Can you share your non sensitive gcs configuration?
**@laxman:** Yeah. GCS path is also there in the above stacktrace.
**@laxman:** ```ERROR [SegmentDeletionManager] [pool-6-thread-1] Had trouble
deleting directories: ```
**@laxman:** Segments are getting uploaded up to gcs cleanly. Also, they are
moved to Deleted_Segments too in gcs. But, they are not getting deleted from
there after the retention period is over
**@laxman:** controller config here (masked some data) ```root@pinot-
controller-0:/opt/pinot# cat /var/pinot/controller/config/pinot-
controller.conf controller.helix.cluster.name=my-views controller.port=9000
controller.zk.str=zookeeper.host.svc.cluster.local:2181/pinot
pinot.set.instance.id.to.hostname=true controller.data.dir=
controller.enable.split.commit=true
controller.local.temp.dir=/var/pinot/controller/data/temp
=org.apache.pinot.plugin.filesystem.GcsPinotFS
pinot.controller.storage.factory.gs.projectId=XXXXXXXXX
pinot.controller.storage.factory.gs.gcpKey=XXXXXXXX.json
pinot.controller.segment.fetcher.protocols=file,http,gs
pinot.controller.segment.fetcher.gs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher```
**@fx19880617:** @elon.azoulay have you seen similar issue or any resolution
is recommended?
**@elon.azoulay:** Checking, I think I know why, will update shortly
**@elon.azoulay:** Working on a fix, will update shortly
**@elon.azoulay:** thanks for catching this @laxman! - we are affected by this
as well
**@fx19880617:** Thanks Elon!
**@laxman:** Thanks for the update @elon.azoulay. Curious to know where this
bug is in the code. Also, is there workaround?
### _#getting-started_
**@hello:** @hello has joined the channel
### _#feat-partial-upsert_
**@yupeng:** @yupeng has joined the channel
**@g.kishore:** @g.kishore has joined the channel
**@jackie.jxt:** @jackie.jxt has joined the channel
**@qiaochu:** @qiaochu has joined the channel
**@tingchen:** @tingchen has joined the channel
**@yupeng:** starting for the design
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org