You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@eventmesh.apache.org by "pandaapo (via GitHub)" <gi...@apache.org> on 2023/12/10 09:31:44 UTC
[I] [Enhancement] Implement the function of File Source Connector (eventmesh)
pandaapo opened a new issue, #4635:
URL: https://github.com/apache/eventmesh/issues/4635
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/eventmesh/issues?q=is%3Aissue) and found no similar issues.
### Enhancement Request
At present, the File Source Connector is basically an empty implementation.
### Describe the solution you'd like
Implement the function of File Source Connector.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) *
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [I] [Enhancement] Implement the function of File Source Connector (eventmesh)
Posted by "HarshSawarkar (via GitHub)" <gi...@apache.org>.
HarshSawarkar commented on issue #4635:
URL: https://github.com/apache/eventmesh/issues/4635#issuecomment-1851390804
Thanks @pandaapo for your suggestion.I have started the implementation already.I am stuck at this **poll()** method.Any suggestions for implementing it would be of great help?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [I] [Enhancement] Implement the function of File Source Connector (eventmesh)
Posted by "HarshSawarkar (via GitHub)" <gi...@apache.org>.
HarshSawarkar commented on issue #4635:
URL: https://github.com/apache/eventmesh/issues/4635#issuecomment-1849708013
Hi @pandaapo I would like to work on this issue.Can I get some information on how to implement (start(),commit(),stop())this methods?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [I] [Enhancement] Implement the function of File Source Connector (eventmesh)
Posted by "Pil0tXia (via GitHub)" <gi...@apache.org>.
Pil0tXia closed issue #4635: [Enhancement] Implement the function of File Source Connector
URL: https://github.com/apache/eventmesh/issues/4635
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [I] [Enhancement] Implement the function of File Source Connector (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on issue #4635:
URL: https://github.com/apache/eventmesh/issues/4635#issuecomment-1854902013
> From what I have read about offset in the context of file source, it can be the location of the byte last read or maybe the line number. What if the file is edited from the line existing previously than offset? For eg:- (offset present at line 91 but the user edited the file at line 22).
It is indeed a complex problem, and I feel that relying solely on offset does not seem to solve the problem of incrementally reading file changes.
确实是个复杂的问题，我觉得这单独靠一个偏移量似乎解决不了增量读取文件变更的问题。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [I] [Enhancement] Implement the function of File Source Connector (eventmesh)
Posted by "HarshSawarkar (via GitHub)" <gi...@apache.org>.
HarshSawarkar commented on issue #4635:
URL: https://github.com/apache/eventmesh/issues/4635#issuecomment-1855558503
Hi @pandaapo,I have created a new PR as per the changes mentioned in earlier PR.Can you please review it again?And suggest changes,if any?Here's the link to new PR https://github.com/apache/eventmesh/pull/4650
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [I] [Enhancement] Implement the function of File Source Connector (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on issue #4635:
URL: https://github.com/apache/eventmesh/issues/4635#issuecomment-1849764971
Welcome!
Firstly, the File Source Connector primarily retrieves content from files, and Java offers various methods for obtaining file content. For example, you can establish an input stream to the target file in the `start()` method, and the `stop()` method naturally involves closing resources that need to be released, such as the previously opened input stream. The `commit()` method is mainly used to record the position of data reading, it is not necessary to implement it for now. However, if you can accurately record the position of the data already read, you can implement this method `commit()`. My suggestion is just a reference, and you are free to choose other, possibly better, implementation approaches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [I] [Enhancement] Implement the function of File Source Connector (eventmesh)
Posted by "HarshSawarkar (via GitHub)" <gi...@apache.org>.
HarshSawarkar commented on issue #4635:
URL: https://github.com/apache/eventmesh/issues/4635#issuecomment-1853731975
Hello @pandaapo , I have submitted a pull request. Could you please review it and provide any suggestions for changes if needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [I] [Enhancement] Implement the function of File Source Connector (eventmesh)
Posted by "VishalMCF (via GitHub)" <gi...@apache.org>.
VishalMCF commented on issue #4635:
URL: https://github.com/apache/eventmesh/issues/4635#issuecomment-1853817079
@pandaapo @HarshSawarkar I have a theoretical question regarding the File Source connector. Hope I get some suggestions or any resources to study related to my question. Because we are not implementing the commit() method, every time a change happens in a file our connector will read the entire file and then push it to the event broker. This seems to be inefficient for now but maybe we can ignore that for now.
But let's say if we want to implement the commit() method how are we going to manage the offset? For each file, we need to store that offset persistently somewhere. From what I have read about offset in the context of file source, it can be the location of the byte last read or maybe the line number. What if the file is edited from the line existing previosuly than offset? For eg:- (offset present at line 91 but the user changed at line 22). I am not sure if my understanding is proper about file source connector but I am trying to refine it. Hope my question is clear enough. Thanks in Advance!!
@pandaapo @HarshSawarkar 我有一个关于文件源连接器的理论问题。 希望我能得到一些建议或任何与我的问题相关的研究资源。 因为我们没有实现 commit() 方法，所以每次文件中发生更改时，我们的连接器都会读取整个文件，然后将其推送到事件代理。 目前这似乎效率低下，但也许我们现在可以忽略它。
但是假设如果我们想实现 commit() 方法，我们将如何管理偏移量？ 对于每个文件，我们需要将该偏移量永久存储在某处。 根据我在文件源上下文中读到的偏移量，它可以是上次读取的字节的位置，也可以是行号。 如果文件是从先前存在的行而不是偏移量编辑的，该怎么办？ 例如：-（偏移量出现在第 91 行，但用户在第 22 行更改）。 我不确定我对文件源连接器的理解是否正确，但我正在尝试完善它。 希望我的问题足够清楚。 提前致谢！！
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [I] [Enhancement] Implement the function of File Source Connector (eventmesh)
Posted by "HarshSawarkar (via GitHub)" <gi...@apache.org>.
HarshSawarkar commented on issue #4635:
URL: https://github.com/apache/eventmesh/issues/4635#issuecomment-1851722834
Thanks for the tips! I'll check out other Source Connectors and the File Sink Connector for reference. I appreciate your input and will keep you posted on the progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [I] [Enhancement] Implement the function of File Source Connector (eventmesh)
Posted by "HarshSawarkar (via GitHub)" <gi...@apache.org>.
HarshSawarkar commented on issue #4635:
URL: https://github.com/apache/eventmesh/issues/4635#issuecomment-1901939513
Hello @pandaapo and @Pil0tXia , I appreciate your help and support! I'm eagerly looking forward to making more contributions. By the way, do you happen to know if EventMesh will be part of GSOC 2024? Additionally, could you share some of the ideas for this year? Thanks a bunch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [I] [Enhancement] Implement the function of File Source Connector (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on issue #4635:
URL: https://github.com/apache/eventmesh/issues/4635#issuecomment-1851711185
I think you can refer to the implementation of other Source Connectors, and also can observe the File Sink Connector to think about functions needed this Connector. The above is what I can provide to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [I] [Enhancement] Implement the function of File Source Connector (eventmesh)
Posted by "Pil0tXia (via GitHub)" <gi...@apache.org>.
Pil0tXia commented on issue #4635:
URL: https://github.com/apache/eventmesh/issues/4635#issuecomment-1901965062
@HarshSawarkar
EventMesh will participate in at least one of GSoC/OSPP/GLCC each year. You don't need to worry too much because the registration period for these three events hasn't started yet. You can subscribe to our issues mailing list, where the subjects will be announced as issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [I] [Enhancement] Implement the function of File Source Connector (eventmesh)
Posted by "HarshSawarkar (via GitHub)" <gi...@apache.org>.
HarshSawarkar commented on issue #4635:
URL: https://github.com/apache/eventmesh/issues/4635#issuecomment-1902019520
@Pil0tXia Thanks for the update! Exciting to hear that EventMesh is participating in GSoC/OSPP/GLCC annually. I'll stay tuned for the registration period and subscribe to the issues mailing list. Appreciate your guidance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org