You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "hkpeaks (via GitHub)" <gi...@apache.org> on 2023/08/02 01:31:17 UTC

[GitHub] [arrow] hkpeaks commented on issue #35688: [Go] How to approach to implement Parquet-Go file format

hkpeaks commented on issue #35688:
URL: https://github.com/apache/arrow/issues/35688#issuecomment-1661330659

   I plan use Apache Arrow compatible python libraries to support building a new ETL Framework (it look like an alternative version of SQL statement which specialized in dealing with file, tcp stream, in-memory table and bigdata).
   
   Example: 
   
   UserDefineFunctionName: SourceFile.ext/ Table / SQL.Table ~ ResultFile.ext /Table / SQL.Table
   | Command: Setting
   
   ExpandFile: Fact.csv ~ 1BillionRows.csv
   | ExpandFactor: 123
   
   JoinScenario1: 1BillionRows.csv ~ Test1Results.csv
   | JoinTable: Quantity, Unit_Price => InnerJoin(Master)Multiply(Amount)
   | OrderBy: Date(D) => CreateFolderLake(Shop)
   | Select: Date,Shop,Style,Product,Quantity,Amount
   
   BuildKeyValueTable: Master.csv ~ KeyValueTable
   | BuildKeyValue: Product, Style
   
   JoinScenario2: 1BillionRows.csv ~ Test2AResults.csv
   | JoinKeyValue: Product, Style => AllMatch(KeyValueTable)
   | AddColumn: Quantity, Unit_Price => Multiply(Amount)
   | Filter: Amount(Float > 50000)
   | GroupBy: Product, Style => Count() Sum(Quantity) Sum(Amount)
   | OrderBy: Shop(A)Product(A)Date(D)
   
   SplitFile: Test1Results.csv ~ FolderLake
   | CreateFolderLake: Shop
   
   FilterFolder: Outbox/FolderLake/S15/*.csv ~ Result-FilterFolderLake.csv
   | Filter: Product(222..888) Style(=F)
   
   ReadSample2View: Outbox/Result-FilterFolderLake.csv ~ SampleTable
   | ReadSample: StartPosition%(0) ByteLength(100000)
   | View
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org