You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "s4iko (via GitHub)" <gi...@apache.org> on 2026/08/28 15:27:57 UTC

[I] Unable to upload existing segments to a table [pinot]

s4iko opened a new issue, #19384:
URL: https://github.com/apache/pinot/issues/19384

   Hello,
   
   I try to upload some old segments produced by an REALTIME table to an OFFLINE table.
   
   It is working as expected using the API as explained here: https://docs.pinot.apache.org/build-with-pinot/ingestion/batch-ingestion/segment-upload#uri-push
   
   But the API is for 1 segment and I want to upload an entire "directory" with many segments. For this a job using Pinot Admin seems to be the solution.
   I tried using "SegmentUriPush" but it does not work because my segment are named "myTable__0__0__20260601T1557Z" with no tar.gz extension.
   But this extension seems to be required when the job is used (this is not the case using the API).
   
   Am I doing something wrong or is it a limitation that can be fixed in a future version ?
   
   I use Azure Data Lake Storage as a deep store.
   
   Thanks for your help.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] Unable to upload existing segments to a table [pinot]

Posted by "s4iko (via GitHub) via commits" <co...@pinot.apache.org>.
s4iko commented on issue #19384:
URL: https://github.com/apache/pinot/issues/19384#issuecomment-6741884618

   Hi @havefunatcode , 
   yes you can send your fix, I didn't get the time to work on it.
   Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] Unable to upload existing segments to a table [pinot]

Posted by "havefunatcode (via GitHub) via commits" <co...@pinot.apache.org>.
havefunatcode commented on issue #19384:
URL: https://github.com/apache/pinot/issues/19384#issuecomment-6649663783

   @s4iko are you still planning to send a fix for this? If not, I'd like to pick it up.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] Unable to upload existing segments to a table [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #19384:
URL: https://github.com/apache/pinot/issues/19384#issuecomment-5455719959

   Do you want to help debug the code? Seems a valid case to me


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] Unable to upload existing segments to a table [pinot]

Posted by "s4iko (via GitHub)" <gi...@apache.org>.
s4iko commented on issue #19384:
URL: https://github.com/apache/pinot/issues/19384#issuecomment-5475227634

   Yep, here is what I saw:
   
   `SegmentUriPushJobRunner` extends `BaseSegmentPushJobRunner`.
   
   The method called to run the job is `BaseSegmentPushJobRunner.run()`.
   In this method, there is a call to `SegmentPushUtils.getSegmentUriToTarPathMap(_outputDirURI, _spec.getPushJobSpec(), _files);` to get the segments to upload via `uploadSegments(segmentsUriToTarPathMap);`.
   This util method return only the URIs ending with `.tar.gz`:
   ```
         if (uri.getPath().endsWith(Constants.TAR_GZ_FILE_EXT)) {
           URI updatedURI = SegmentPushUtils.generateSegmentTarURI(outputDirURI, uri, pushSpec.getSegmentUriPrefix(),
               pushSpec.getSegmentUriSuffix());
           segmentUriToTarPathMap.put(updatedURI.toString(), file);
         }
   ```
   
   Ref:
   - https://github.com/apache/pinot/blob/5e914c979b355c0e8c9fd4cd6f401f5f899594b5/pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-standalone/src/main/java/org/apache/pinot/plugin/ingestion/batch/standalone/SegmentUriPushJobRunner.java#L30
   - https://github.com/apache/pinot/blob/5e914c979b355c0e8c9fd4cd6f401f5f899594b5/pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-common/src/main/java/org/apache/pinot/plugin/ingestion/batch/common/BaseSegmentPushJobRunner.java#L138
   - https://github.com/apache/pinot/blob/5e914c979b355c0e8c9fd4cd6f401f5f899594b5/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/SegmentPushUtils.java#L651
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] Unable to upload existing segments to a table [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #19384:
URL: https://github.com/apache/pinot/issues/19384#issuecomment-5500145885

   This might be too restricted. Do you want to help contribute a fix?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] Unable to upload existing segments to a table [pinot]

Posted by "havefunatcode (via GitHub) via commits" <co...@pinot.apache.org>.
havefunatcode commented on issue #19384:
URL: https://github.com/apache/pinot/issues/19384#issuecomment-6750848654

   Thanks @s4iko! I already have a PR up for this: #20160. It adds an opt-in `allowSegmentsWithoutArchiveExtension` flag to the URI push job spec, so the default behavior doesn't change.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org