You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@devlake.apache.org by "narrowizard (via GitHub)" <gi...@apache.org> on 2023/02/03 03:36:42 UTC
[GitHub] [incubator-devlake] narrowizard opened a new issue, #3749: [Bug][e2e] nil support in csv
narrowizard opened a new issue, #3749:
URL: https://github.com/apache/incubator-devlake/issues/3749
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar issues.
### What happened
I have wrote some e2e test with e2ehelper.DataFlowTester, it's a great experience.
But it reports error when my snapshot data is with nil column value.
### What do you expect to happen
As a helper, `DataFlowTester` should support nil value properly.
I noticed that `ImportCsvIntoTabler` will convert empty string to nil value before write to db. It is not a perfect solution as it confused empty string and nil value.
https://github.com/apache/incubator-devlake/blob/fb6c5e2e6131cd8c868183430b57a84e9c89c827/helpers/e2ehelper/data_flow_tester.go#L145-L147
> we always have columns with nil value, and requrement to validate these data.
### How to reproduce
Refer to `What happened` section
### Anything else
The problem is that csv file cannot distinct empty string and nil value.
A solution is that we can use a placeholder string to represent nil value. (maybe a `<nil value>`)
When read from csv file, convert the `<nil value>` to nil. And write a `<nil value>` when writing a nil value.
### Version
main
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] github-actions[bot] commented on issue #3749: [Bug][e2e] nil support in csv
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #3749:
URL: https://github.com/apache/incubator-devlake/issues/3749#issuecomment-1455262005
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] likyh closed issue #3749: [Bug][e2e] nil support in csv
Posted by "likyh (via GitHub)" <gi...@apache.org>.
likyh closed issue #3749: [Bug][e2e] nil support in csv
URL: https://github.com/apache/incubator-devlake/issues/3749
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] github-actions[bot] commented on issue #3749: [Bug][e2e] nil support in csv
Posted by github-actions.
github-actions[bot] commented on issue #3749:
URL: https://github.com/apache/incubator-devlake/issues/3749#issuecomment-1405847075
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] github-actions[bot] commented on issue #3749: [Bug][e2e] nil support in csv
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #3749:
URL: https://github.com/apache/incubator-devlake/issues/3749#issuecomment-1414542092
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] github-actions[bot] commented on issue #3749: [Bug][e2e] nil support in csv
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #3749:
URL: https://github.com/apache/incubator-devlake/issues/3749#issuecomment-1364418503
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh closed issue #3749: [Bug][e2e] nil support in csv
Posted by GitBox <gi...@apache.org>.
klesh closed issue #3749: [Bug][e2e] nil support in csv
URL: https://github.com/apache/incubator-devlake/issues/3749
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] github-actions[bot] closed issue #3749: [Bug][e2e] nil support in csv
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #3749: [Bug][e2e] nil support in csv
URL: https://github.com/apache/incubator-devlake/issues/3749
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] Startrekzky commented on issue #3749: [Bug][e2e] nil support in csv
Posted by "Startrekzky (via GitHub)" <gi...@apache.org>.
Startrekzky commented on issue #3749:
URL: https://github.com/apache/incubator-devlake/issues/3749#issuecomment-1456225994
@mindlesscloud What's the progress of this bug?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] mappjzc commented on issue #3749: [Bug][e2e] nil support in csv
Posted by GitBox <gi...@apache.org>.
mappjzc commented on issue #3749:
URL: https://github.com/apache/incubator-devlake/issues/3749#issuecomment-1318490967
Thanks for you issue, are you willing to submit PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-devlake] klesh commented on issue #3749: [Bug][e2e] nil support in csv
Posted by GitBox <gi...@apache.org>.
klesh commented on issue #3749:
URL: https://github.com/apache/incubator-devlake/issues/3749#issuecomment-1324738931
Found an practical solution from https://docs.snowflake.com/en/user-guide/data-unload-considerations.html#:~:text=Empty%20Strings%20and%20NULL%20Values,-An%20empty%20string&text=In%20CSV%20files%2C%20a%20NULL,null%20)%20or%20any%20unique%20string.
That is to quote all string with double quote, and empty string without quote represents `nil`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@devlake.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org