[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #281: Remove heap sized limit for parsing

[GitBox <gi...@apache.org>]

stevedlawrence commented on a change in pull request #281: Remove heap sized limit for parsing
URL: https://github.com/apache/incubator-daffodil/pull/281#discussion_r342530553
 
 

 ##########
 File path: daffodil-cli/src/it/resources/org/apache/daffodil/CLI/input/gen_blob.py
 ##########
 @@ -0,0 +1,18 @@
+#! /usr/bin/python
+
+import argparse
+import binascii
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("-s", "--size", help="Specify the size of the generated file in MB", type=int)
+    parser.add_argument("-o", "--output", help="Path to the file that will be generated")
+    args = parser.parse_args()
+
+    bytes_written = 0
+    with open(args.output, 'wb') as fout:
+        while (bytes_written < (args.size * 1024000)):
+            fout.write(binascii.unhexlify('deadbeef'))
+            bytes_written += 4
+
+    print("Wrote %d bytes" % bytes_written)
 
 Review comment:
   Was this used to generate 2_01GB.zip? In general, Apache reccommends that anything that is generated not be included in a repo, but instead include instructions on how to generate it. Either way a user needs to perform an action to get the 2GB file (either run this script or unzip a file), so I'd rather them just run this script.
   
   Another though, it might make for a more interesting test if the first 8 bytes were an unsigned long of the number of bytes. That's more realistic to how most blob files are going to look, and it allows the schema to have outputValueCalc for unparse and runtime calculated lengths parse. And then you can have a single schema that could parse anything that this scrip could generate without having to modify the schema.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services