Re: [I] [VL] Improve the implementation of Spark function input_file_name [incubator-gluten]

["zhztheplayer (via GitHub)" <gi...@apache.org>]

zhztheplayer commented on issue #6157:
URL: https://github.com/apache/incubator-gluten/issues/6157#issuecomment-2182011687

   > hey @zhztheplayer I thought a while and zhen's suggestion to convert `input_file_name` to `partition column` maybe a good workaround which don't need we update scan node but it is actually not semantic right, since file_name is actually more metadata column not partition column.
   > 
   > But it help me bring another idea which can overcome the above shortcomings.
   > 
   > The old implemented way need a fallback rule due to it introduce a new metadta column named `$xxx$` which vanilla spark can't recognize so need be revert using fallback strategy, however actually vanilla spark has it self's metadata column named `_metadata.file_name` which has same semantic as `input_file_name`, so we can actually just leverage one simple rule to convert the `input_file_name` to `_metadata.file_name` which works in both vinalla spark and native.
   > 
   > With that, we only need one rule to keep it simple and work.
   > 
   > for example
   > 
   > ```
   > Project [input_file_name() AS input_file_name()#7, id#0]
   > +- Relation [id#0] parquet
   > ```
   > 
   > would be
   > 
   > ```
   > Project [_metadata#6.file_name AS file_name#7, id#0]
   > +- Relation [id#0,_metadata#6] parquet
   > ```
   > 
   > I will try locally to verify if the idea works. But let me know if you think that way is acceptable
   > 
   > @zhztheplayer / @FelixYBW / @zhli1142015 FYI
   
   That sounds great if it works.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org