You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "hkpeaks (via GitHub)" <gi...@apache.org> on 2023/08/02 01:31:17 UTC
[GitHub] [arrow] hkpeaks commented on issue #35688: [Go] How to approach to implement Parquet-Go file format
hkpeaks commented on issue #35688:
URL: https://github.com/apache/arrow/issues/35688#issuecomment-1661330659
I plan use Apache Arrow compatible python libraries to support building a new ETL Framework (it look like an alternative version of SQL statement which specialized in dealing with file, tcp stream, in-memory table and bigdata).
Example:
UserDefineFunctionName: SourceFile.ext/ Table / SQL.Table ~ ResultFile.ext /Table / SQL.Table
| Command: Setting
ExpandFile: Fact.csv ~ 1BillionRows.csv
| ExpandFactor: 123
JoinScenario1: 1BillionRows.csv ~ Test1Results.csv
| JoinTable: Quantity, Unit_Price => InnerJoin(Master)Multiply(Amount)
| OrderBy: Date(D) => CreateFolderLake(Shop)
| Select: Date,Shop,Style,Product,Quantity,Amount
BuildKeyValueTable: Master.csv ~ KeyValueTable
| BuildKeyValue: Product, Style
JoinScenario2: 1BillionRows.csv ~ Test2AResults.csv
| JoinKeyValue: Product, Style => AllMatch(KeyValueTable)
| AddColumn: Quantity, Unit_Price => Multiply(Amount)
| Filter: Amount(Float > 50000)
| GroupBy: Product, Style => Count() Sum(Quantity) Sum(Amount)
| OrderBy: Shop(A)Product(A)Date(D)
SplitFile: Test1Results.csv ~ FolderLake
| CreateFolderLake: Shop
FilterFolder: Outbox/FolderLake/S15/*.csv ~ Result-FilterFolderLake.csv
| Filter: Product(222..888) Style(=F)
ReadSample2View: Outbox/Result-FilterFolderLake.csv ~ SampleTable
| ReadSample: StartPosition%(0) ByteLength(100000)
| View
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org