You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@streampipes.apache.org by "tenthe (via GitHub)" <gi...@apache.org> on 2023/12/14 16:01:49 UTC

[I] Dealing with files in the File Stream Adapter that don't have a timestamp on every line.

tenthe opened a new issue, #2338:
URL: https://github.com/apache/streampipes/issues/2338

   ### Discussed in https://github.com/apache/streampipes/discussions/2298
   
   <div type='discussions-op-text'>
   
   <sup>Originally posted by **AkthemRehab** December  8, 2023</sup>
   Hello,
   I am wondering if the file stream has an option to give a unique timestamp for each line? An example use case is a vibration reading that usually has 'one' original timestamp. Usually the name of the file is the timestamp and the readings within the file are taken within the span of 1 second or less.
   
   Best,
   Akthem</div>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@streampipes.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Dealing with files in the File Stream Adapter that don't have a timestamp on every line.

Posted by "tenthe (via GitHub)" <gi...@apache.org>.
tenthe commented on issue #2338:
URL: https://github.com/apache/streampipes/issues/2338#issuecomment-4221934999

   I attempted to support both cases:
   
   * files without a timestamp
   * files with a string-based timestamp
   
   However, implementing this is not straightforward. It would require the adapter to parse events during configuration, which adds complexity. This added complexity would affect all use cases, even though it is only needed for these two edge cases, making it less practical overall.
   
   Because of this, the recommended approach is to preprocess the replay file before using the adapter in these two edge cases.
   
   If the file already contains a Unix timestamp column, everything continues to work as expected.
   
   Additionally, PR #4366 introduces a startup validation in the adapter to ensure that the input data contains a valid timestamp value.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@streampipes.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Dealing with files in the File Stream Adapter that don't have a timestamp on every line.

Posted by "Jayyadav1903 (via GitHub)" <gi...@apache.org>.
Jayyadav1903 commented on issue #2338:
URL: https://github.com/apache/streampipes/issues/2338#issuecomment-3882901730

   Hi @tenthe,
   
   I’d like to work on this. My proposal is to introduce a configurable timestamp strategy in the File Stream Adapter, while keeping existing behavior unchanged.
   
   For an initial PR, I plan to:
   
   Preserve the current default behavior
   
   Add a FILE_NAME_TIMESTAMP strategy that extracts a timestamp from the file name and applies it to all lines in the file
   
   This would directly address the original use case while keeping the change minimal and backward-compatible.
   
   Let me know if this direction aligns with expectations before I start implementing.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@streampipes.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Dealing with files in the File Stream Adapter that don't have a timestamp on every line.

Posted by "tenthe (via GitHub)" <gi...@apache.org>.
tenthe commented on issue #2338:
URL: https://github.com/apache/streampipes/issues/2338#issuecomment-4212434546

   There is currently an issue when the timestamp in the original CSV is provided as a string that needs to be parsed.
   
   The root of the problem lies in `FileReplayAdapter.getTimestampFromEvent`, where the timestamp must be extracted from the original event. This extraction happens **before** the transformation script is executed. However, we need access to the timestamp at this stage because it may be required to adjust the replay speed.
   
   ### Problem to solve
   
   We need a consistent way to handle timestamps at this point in the adapter.
   
   #### Option 1: Enforce a strict format
   
   * Define stricter requirements for the file replay adapter.
   * Require each line to contain a Unix timestamp.
   * If the requirement is not met, an error is shown to the user.
   
   #### Option 2: Parse string timestamps
   
   * Convert string-based timestamps into Unix timestamps (`long`).
   * Challenge: In Java, parsing requires a known timestamp pattern.
   * Unlike TypeScript (`new Date(event.timestamp).getTime()`), this cannot be handled generically without additional information.
   
   To reproduce and test this issue, the e2e test `valueRules.spec.ts` can be executed. This test is currently failing.
   
   ### Additional open issue
   
   There is also the case where no timestamp is present in the event stream at all.
   This scenario still needs a solution as well.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@streampipes.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Dealing with files in the File Stream Adapter that don't have a timestamp on every line.

Posted by "tenthe (via GitHub)" <gi...@apache.org>.
tenthe closed issue #2338: Dealing with files in the File Stream Adapter that don't have a timestamp on every line.
URL: https://github.com/apache/streampipes/issues/2338


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@streampipes.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org