You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/08/13 16:33:28 UTC

[GitHub] [iceberg] amogh-jahagirdar opened a new pull request, #5516: API: Add Generate Symlink Manifest API

amogh-jahagirdar opened a new pull request, #5516:
URL: https://github.com/apache/iceberg/pull/5516

   This change adds the API definition for Generating Symlink Manifest. Separating API PR from https://github.com/apache/iceberg/pull/5398  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] kbendick commented on pull request #5516: API: Add Generate Symlink Manifest API

Posted by GitBox <gi...@apache.org>.
kbendick commented on PR #5516:
URL: https://github.com/apache/iceberg/pull/5516#issuecomment-1216075644

   > While there are definitely caveats like renaming column name case and presence of V2 delete files that we should warn about, I also agree with @jackye1995 , generate SymlinkManifest seems quite useful, we have seen some asks for interop of data in iceberg table to non-Iceberg systems (in house file-based tools for instance).
   > 
   > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/SymlinkTextInputFormat.java seems a pretty standard way to provide file listing, and both generating/reading seems supported in a variety of data warehouse systems.
   
   I strongly share @rdblue’s concerns about properly reading deletes and can see potential concerns about schema evolution, but I do have seen that symlink text input format is pretty useful for interop and _usually, in my experience_ it’s (re)loaded every time it’s needed. But that might not be the common case. Does it have to be rebuilt entirely every time?
   
   I think it has a lot of value for interop with specific BI tools that are used for presentations etc that the end user really doesn’t have a choice on.
   
   In my experience, I’ve seen this needed especially for specific one off things like financial report demonstrations where the BI tool is for some reason set in stone.
   
   I’m not familiar enough with the format and concerns with schema evolution — again, usually my usage of it has been one off or rebuild / reload on every “refresh” — but I do think this would be good for Iceberg overall.
   
   But ensuring schema and properly applied deletes on every creation should be done with a lot of care.
   
   At least for BI tools that wouldn’t support a custom format at all (as this is much less efficient than reading the Iceberg table directly). I can share concern with not discouraging other tools from skipping on supporting Iceberg natively. But maybe this will provide those tools more of an incentive in the long run.
   
   My 2 cents on the matter. 100% happy to learn more about the format’s specifics to help support it if need be. 🙂 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] kbendick commented on a diff in pull request #5516: API: Add Generate Symlink Manifest API

Posted by GitBox <gi...@apache.org>.
kbendick commented on code in PR #5516:
URL: https://github.com/apache/iceberg/pull/5516#discussion_r945209431


##########
api/src/main/java/org/apache/iceberg/actions/GenerateSymlinkManifest.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+/**
+ * An action that generates a symlink manifest table for an Iceberg table.
+ *
+ * <p>This action will generate symlink manifests for a table. A symlink manifest is a file with
+ * links to the data files in the table. Systems which do not integrate with Iceberg can read the

Review Comment:
   Nit: This paragraph is correct in explaining what a symlink manifest is, but I do encounter a number of people on Slack / in the community who aren’t aware that Icebergs metadata tracks the files already. I worry this will make that _less_ clear.
   
   Is it possible to document the symlink manifest without making it seem like Iceberg doesn’t already track the files? Maybe something like the following would help.
   
   `A symlink manifest is a standardized table format (specification file?) that many other systems can read. It consists of links to the data files in the table, like Iceberg, but without the additional metadata that Iceberg includes. Systems which do not integrate with Iceberg can read the …`
   
   Not necessarily in love with that phrasing, but given how often we wind up fielding questions from users about performance due to table layout etc because newer users aren’t always aware that Iceberg tracks the files in its manifests, it would be nice to make that a little more explicit somehow so as not to potentially introduce more confusion. Again, not blocking as this is a good description but I know we both answered a question about table layout on Slack just this week for a user who was assuming that Iceberg was doing listing to get partition info. It would be good to reduce the chances of getting more of these questions if possible. 🙂 
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on pull request #5516: API: Add Generate Symlink Manifest API

Posted by GitBox <gi...@apache.org>.
rdblue commented on PR #5516:
URL: https://github.com/apache/iceberg/pull/5516#issuecomment-1214418013

   @jackye1995, @amogh-jahagirdar, is this a good idea? I'm skeptical. If the aim is to have Redshift support, then I think Redshift should implement a way to call one of the APIs that can perform job planning instead.
   
   I find it unlikely that readers that do not support Iceberg tables will be able to read the files correctly and apply the schema evolution rules. I don't like the idea of making it easy for external systems to claim to support Iceberg tables via some export mechanism like this, but then give incorrect results because they're just reading Parquet files.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] kbendick commented on a diff in pull request #5516: API: Add Generate Symlink Manifest API

Posted by GitBox <gi...@apache.org>.
kbendick commented on code in PR #5516:
URL: https://github.com/apache/iceberg/pull/5516#discussion_r945209431


##########
api/src/main/java/org/apache/iceberg/actions/GenerateSymlinkManifest.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+/**
+ * An action that generates a symlink manifest table for an Iceberg table.
+ *
+ * <p>This action will generate symlink manifests for a table. A symlink manifest is a file with
+ * links to the data files in the table. Systems which do not integrate with Iceberg can read the

Review Comment:
   Nit: This paragraph is correct in explaining what a symlink manifest is, but I do encounter a number of people on Slack / in the community who aren’t aware that Icebergs metadata tracks the files already. I worry this will make that _less_ clear.
   
   Is it possible to document the symlink manifest without making it seem like Iceberg doesn’t already track the files? Maybe something like the following would help.
   
   `A symlink manifest is a standardized table format (specification file?) that many other systems can read. It consists of links to the data files in the table, like Iceberg, but without the additional metadata that Iceberg includes. Systems which do not integrate with Iceberg can read the …`
   
   Not necessarily in love with the phrasing I provided here. But given that we often wind up fielding questions from users because they don’t recognize that Iceberg tracks all table files explicitly, it would be nice to make that a bit more explicit when mentioning the symlink manifest format so as not to potentially introduce more confusion (especially for users who don’t necessarily read the docs, but might have questions on this action e.g. with Athena and then _do_ read this JavaDoc ).
   
   Again, not blocking as this is a good description overall. But I know we both answered a question about table layout on Slack just this week for a user who was assuming that Iceberg was doing listing to get partition info. It would be good to reduce the chances of getting more of these types of questions if possible. 🙂 
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a diff in pull request #5516: API: Add Generate Symlink Manifest API

Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #5516:
URL: https://github.com/apache/iceberg/pull/5516#discussion_r945319257


##########
api/src/main/java/org/apache/iceberg/actions/GenerateSymlinkManifest.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+/**
+ * An action that generates a symlink manifest table for an Iceberg table.
+ *
+ * <p>This action will generate symlink manifests for a table. A symlink manifest is a file with
+ * links to the data files in the table. Systems which do not integrate with Iceberg can read the
+ * underlying data in the Iceberg tables as an external table. The symlink manifest table will
+ * expose all the hidden partitions as the union of historical partition specs.
+ */
+public interface GenerateSymlinkManifest
+    extends Action<GenerateSymlinkManifest, GenerateSymlinkManifest.Result> {
+
+  /**
+   * Set a root location for the symlink manifest table. If a root location is not provided the
+   * default location will be at table_root/_symlink_format_manifest/snapshot_id/
+   *
+   * @param rootLocation root location for symlink manifest table
+   * @return this for method chaining
+   */
+  GenerateSymlinkManifest rootLocation(String rootLocation);
+
+  /** The action result that contains a summary of the execution. */
+  interface Result {

Review Comment:
   Why does this not include the manifest location?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on pull request #5516: API: Add Generate Symlink Manifest API

Posted by GitBox <gi...@apache.org>.
rdblue commented on PR #5516:
URL: https://github.com/apache/iceberg/pull/5516#issuecomment-1222838331

   I don't think that giving a best-effort dump of data files is a good idea for Iceberg to do. Someone could certainly write a tool that does this, but I think would be bad for Iceberg to produce a manifest file that may or may not be correct.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #5516: API: Add Generate Symlink Manifest API

Posted by GitBox <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #5516:
URL: https://github.com/apache/iceberg/pull/5516#discussion_r945985691


##########
api/src/main/java/org/apache/iceberg/actions/GenerateSymlinkManifest.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+/**
+ * An action that generates a symlink manifest table for an Iceberg table.
+ *
+ * <p>This action will generate symlink manifests for a table. A symlink manifest is a file with
+ * links to the data files in the table. Systems which do not integrate with Iceberg can read the
+ * underlying data in the Iceberg tables as an external table. The symlink manifest table will
+ * expose all the hidden partitions as the union of historical partition specs.
+ */
+public interface GenerateSymlinkManifest
+    extends Action<GenerateSymlinkManifest, GenerateSymlinkManifest.Result> {
+
+  /**
+   * Set a root location for the symlink manifest table. If a root location is not provided the
+   * default location will be at table_root/_symlink_format_manifest/snapshot_id/
+   *
+   * @param rootLocation root location for symlink manifest table
+   * @return this for method chaining
+   */
+  GenerateSymlinkManifest rootLocation(String rootLocation);

Review Comment:
   Actually...after more thought I don't really know what a user would use it for other than serving as the root for an external table location. So I think rootLocation still makes sense. Maybe outputRootLocation makes it more obvious that it's the root location for the generated table? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] kbendick commented on a diff in pull request #5516: API: Add Generate Symlink Manifest API

Posted by GitBox <gi...@apache.org>.
kbendick commented on code in PR #5516:
URL: https://github.com/apache/iceberg/pull/5516#discussion_r945208910


##########
api/src/main/java/org/apache/iceberg/actions/GenerateSymlinkManifest.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+/**
+ * An action that generates a symlink manifest table for an Iceberg table.
+ *
+ * <p>This action will generate symlink manifests for a table. A symlink manifest is a file with
+ * links to the data files in the table. Systems which do not integrate with Iceberg can read the
+ * underlying data in the Iceberg tables as an external table. The symlink manifest table will
+ * expose all the hidden partitions as the union of historical partition specs.
+ */
+public interface GenerateSymlinkManifest
+    extends Action<GenerateSymlinkManifest, GenerateSymlinkManifest.Result> {
+
+  /**
+   * Set a root location for the symlink manifest table. If a root location is not provided the
+   * default location will be at table_root/_symlink_format_manifest/snapshot_id/
+   *
+   * @param rootLocation root location for symlink manifest table
+   * @return this for method chaining
+   */
+  GenerateSymlinkManifest rootLocation(String rootLocation);

Review Comment:
   Nit / non-blocking:
   
   1) From reading the JavaDoc, I'm not sure if the `rootLocation` function is the prefix and then we'll still append `/${snapshot_id}/`.
   
   2) Arguably that's an implementation detail but But I’m also not sure how `rootLocation` should be passed in. For example, for a table at `s3://bucket/database/table`, if I wanted the symlink manifest to be at `s3://bucket/database/table/current_symlink/`, what would I need to pass in?
   
   It’s possible that both of these are more of a question for the implementing class, but at least for the second question it would be nice when using Iceberg as a library if an example `rootLocation` and resulting path or something could be given. Like should it include the URI (s3://)? And will `snapshot_Id` still be appended after that? Some of these questions might be implementation specific, but it would be nice if the javadoc could clear up at least the expectated format of `rootLocation` without having to consult unit tests etc.
   
   But I also recognize that there might be more javadocs to come (like for the corresponding Spark action) and some of those questions might be better answered elsewhere.
   
   Not blocking but something to think about  🙂 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #5516: API: Add Generate Symlink Manifest API

Posted by GitBox <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #5516:
URL: https://github.com/apache/iceberg/pull/5516#discussion_r945985691


##########
api/src/main/java/org/apache/iceberg/actions/GenerateSymlinkManifest.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+/**
+ * An action that generates a symlink manifest table for an Iceberg table.
+ *
+ * <p>This action will generate symlink manifests for a table. A symlink manifest is a file with
+ * links to the data files in the table. Systems which do not integrate with Iceberg can read the
+ * underlying data in the Iceberg tables as an external table. The symlink manifest table will
+ * expose all the hidden partitions as the union of historical partition specs.
+ */
+public interface GenerateSymlinkManifest
+    extends Action<GenerateSymlinkManifest, GenerateSymlinkManifest.Result> {
+
+  /**
+   * Set a root location for the symlink manifest table. If a root location is not provided the
+   * default location will be at table_root/_symlink_format_manifest/snapshot_id/
+   *
+   * @param rootLocation root location for symlink manifest table
+   * @return this for method chaining
+   */
+  GenerateSymlinkManifest rootLocation(String rootLocation);

Review Comment:
   Actually...after more thought I don't really know what a user would use it for other than serving as the root for an external table location. So I think rootLocation still makes sense. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] szehon-ho commented on pull request #5516: API: Add Generate Symlink Manifest API

Posted by GitBox <gi...@apache.org>.
szehon-ho commented on PR #5516:
URL: https://github.com/apache/iceberg/pull/5516#issuecomment-1218383697

   Hi guys, just continuing on the discussion from the sync (where there was not consensus to proceed on this)
   
   I was thinking for these kind of use cases , Iceberg could potentially offer an 'export' functionality that writes either to a symlink manifest or just directory-based table. Then we could short-circuit and re-use existing files if there are no schema evolution, delete files, and maybe default values defined, or just do a rewrite if there are. Thinking it could be a nice kind of a nice counterpart to 'import', and also allow users who are primariily interested in the base features of Iceberg but not necessarily schema-evolution or merge-on-read a nice interop with non-Iceberg environments.  @rdblue wondering if that makes more sense?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] jackye1995 commented on pull request #5516: API: Add Generate Symlink Manifest API

Posted by GitBox <gi...@apache.org>.
jackye1995 commented on PR #5516:
URL: https://github.com/apache/iceberg/pull/5516#issuecomment-1215502579

   Yes the original PR I worked was for a temporary support for Redshift, that's why I did not even bother merging the PR. However we started to realize that symlink manifest is a way for many data warehouse systems to integrate with new products before knowing details and offering native support, and it seems like a beneficial feature to have in general. For example: https://www.vertica.com/kb/Vertica_DeltaLake_Technical_Exploration/Content/Partner/Vertica_DeltaLake_Technical_Exploration.htm. That is why Amogh start to pick up the work and make it an actual API in Iceberg. @rdblue any thoughts?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] szehon-ho commented on pull request #5516: API: Add Generate Symlink Manifest API

Posted by GitBox <gi...@apache.org>.
szehon-ho commented on PR #5516:
URL: https://github.com/apache/iceberg/pull/5516#issuecomment-1216068428

   While there are definitely caveats like renaming column name case and presence of V2 delete files that we should warn about, I also agree with @jackye1995 , generate SymlinkManifest seems quite useful, we have seen some asks for interop of data in iceberg table to non-Iceberg systems (in house file-based tools for instance).   
   
   https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/SymlinkTextInputFormat.java seems a pretty standard way to provide file listing, and both generating/reading seems supported in a variety of data warehouse systems.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] jackye1995 commented on pull request #5516: API: Add Generate Symlink Manifest API

Posted by GitBox <gi...@apache.org>.
jackye1995 commented on PR #5516:
URL: https://github.com/apache/iceberg/pull/5516#issuecomment-1217102663

   The assumptions I have with the symlink manifest are basically:
   1. this is only for reading a set of Parquet/ORC/Avro files in a set of manifest list with some basic partitioning support, no any other operation support like schema evolution
   2. the result table does not have latest merge-on-read deletes, basically a read-optimized view if we want to use the Hudi term
   
   we should make that very clear if we offer the feature.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a diff in pull request #5516: API: Add Generate Symlink Manifest API

Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #5516:
URL: https://github.com/apache/iceberg/pull/5516#discussion_r945319374


##########
api/src/main/java/org/apache/iceberg/actions/GenerateSymlinkManifest.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+/**
+ * An action that generates a symlink manifest table for an Iceberg table.
+ *
+ * <p>This action will generate symlink manifests for a table. A symlink manifest is a file with
+ * links to the data files in the table. Systems which do not integrate with Iceberg can read the
+ * underlying data in the Iceberg tables as an external table. The symlink manifest table will
+ * expose all the hidden partitions as the union of historical partition specs.
+ */
+public interface GenerateSymlinkManifest
+    extends Action<GenerateSymlinkManifest, GenerateSymlinkManifest.Result> {
+
+  /**
+   * Set a root location for the symlink manifest table. If a root location is not provided the
+   * default location will be at table_root/_symlink_format_manifest/snapshot_id/
+   *
+   * @param rootLocation root location for symlink manifest table
+   * @return this for method chaining
+   */
+  GenerateSymlinkManifest rootLocation(String rootLocation);

Review Comment:
   What is `rootLocation`?
   
   It isn't clear what you mean by "root" here. Is that a table root? Or is it just an output location? If it's the latter, then `outputLocation` is a better name.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #5516: API: Add Generate Symlink Manifest API

Posted by GitBox <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #5516:
URL: https://github.com/apache/iceberg/pull/5516#discussion_r945323216


##########
api/src/main/java/org/apache/iceberg/actions/GenerateSymlinkManifest.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+/**
+ * An action that generates a symlink manifest table for an Iceberg table.
+ *
+ * <p>This action will generate symlink manifests for a table. A symlink manifest is a file with
+ * links to the data files in the table. Systems which do not integrate with Iceberg can read the
+ * underlying data in the Iceberg tables as an external table. The symlink manifest table will
+ * expose all the hidden partitions as the union of historical partition specs.
+ */
+public interface GenerateSymlinkManifest
+    extends Action<GenerateSymlinkManifest, GenerateSymlinkManifest.Result> {
+
+  /**
+   * Set a root location for the symlink manifest table. If a root location is not provided the
+   * default location will be at table_root/_symlink_format_manifest/snapshot_id/
+   *
+   * @param rootLocation root location for symlink manifest table
+   * @return this for method chaining
+   */
+  GenerateSymlinkManifest rootLocation(String rootLocation);

Review Comment:
   Good point, it's an output location for the symlink manifest. Earlier we called it symlink manifest root location because it would serve as the root table location for a user who would create an external table with this as the root. But I agree it's odd to name the API on what a user may or may not use it for. outputLocation is better.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] API: Add Generate Symlink Manifest API [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5516:
URL: https://github.com/apache/iceberg/pull/5516#issuecomment-2294473827

   This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] API: Add Generate Symlink Manifest API [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #5516: API: Add Generate Symlink Manifest API
URL: https://github.com/apache/iceberg/pull/5516


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] API: Add Generate Symlink Manifest API [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5516:
URL: https://github.com/apache/iceberg/pull/5516#issuecomment-2309064339

   This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org