You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "karim-ramadan (via GitHub)" <gi...@apache.org> on 2023/04/06 07:25:50 UTC
[GitHub] [iceberg] karim-ramadan opened a new issue, #7287: Broken Unit tests on Windows OS
karim-ramadan opened a new issue, #7287:
URL: https://github.com/apache/iceberg/issues/7287
### Apache Iceberg version
main (development)
### Query engine
None
### Please describe the bug 🐞
### Problem statement
Most of the Hadoop/Spark files API use URIs to address resource locations, this solution should work for any OS, but due to some manual URI generation inside of unit tests such as:
`String location = table.table().location().replaceFirst("file:", "");
new File(location + "/data/trashfile").createNewFile();`
and some direct usage of java.io.File instead of URI:
`sql("ALTER TABLE %s ADD PARTITION(id=0) LOCATION '%s'", source, temp.newFolder().toString());`
At the current state, unit tests work only on Unix-based operating systems.
### Problem solution
Replace all manual URI generation with Java API and always use URIs when specifying locations.
Here is the solution to the 2 examples:
`String location = table.table().location();
File trashFile = new File(Paths.get(URI.create(location)).toFile(), "/data/trashfile");
trashFile.createNewFile();
URI trashUri = trashFile.toURI();`
`sql("ALTER TABLE %s ADD PARTITION(id=0) LOCATION '%s'", source, temp.newFolder().toURI().toString());`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Broken Unit tests on Windows OS [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7287:
URL: https://github.com/apache/iceberg/issues/7287#issuecomment-1858628410
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] karim-ramadan commented on issue #7287: Broken Unit tests on Windows OS
Posted by "karim-ramadan (via GitHub)" <gi...@apache.org>.
karim-ramadan commented on issue #7287:
URL: https://github.com/apache/iceberg/issues/7287#issuecomment-1500047391
Hi @jackye1995 I'd like to open a PR to fix it, I was also thinking instead of fixing all the problems case by case we could create a class like this
`
public class UriFileSystem {
File parent;
public UriFileSystem(URI uri) {
this.parent = Paths.get(uri).toFile();
}
public UriFileSystem(File file) {
this.parent = file;
}
public UriFileSystem(Path path) {
this.parent = path.toFile();
}
public URI newFolder(String... paths) throws IOException {
if (paths.length == 0) {
return parent.toURI();
}
File f = parent;
for (String p : paths) {
f = new File(f, p);
}
f.mkdirs();
return f.toURI();
}
public URI newFile(String name) throws IOException {
File f = new File(parent, name);
f.createNewFile();
return f.toURI();
}
public static Stream<URI> listFiles(URI location) {
return Arrays.stream(Objects.requireNonNull(Paths.get(location).toFile().listFiles()))
.map(File::toURI);
}
}
`
Inside the core test module, in this way, we could manage all resources as URIs avoiding silly mistakes and ensuring that in the future tests are always compatible across all OS.
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Broken Unit tests on Windows OS [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7287:
URL: https://github.com/apache/iceberg/issues/7287#issuecomment-1763216102
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Broken Unit tests on Windows OS [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #7287: Broken Unit tests on Windows OS
URL: https://github.com/apache/iceberg/issues/7287
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on issue #7287: Broken Unit tests on Windows OS
Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on issue #7287:
URL: https://github.com/apache/iceberg/issues/7287#issuecomment-1499595322
Thanks for bringing this up! Windows compatibility is a known issue that we never get time to address. Are you interested in working on this, or are you asking for the community to address it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] karim-ramadan commented on issue #7287: Broken Unit tests on Windows OS
Posted by "karim-ramadan (via GitHub)" <gi...@apache.org>.
karim-ramadan commented on issue #7287:
URL: https://github.com/apache/iceberg/issues/7287#issuecomment-1511538613
Hi, @jackye1995 could you give me feedback on this? would you value a contribution like this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org