You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@flex.apache.org by bilbosax <wa...@comcast.net> on 2016/08/27 00:50:29 UTC
Downloading Large Amounts of Data from Database
I am running into an issue that I believe is going to take some creative
problem solving. I have created a service to download large amounts of data
from a MySQL database, at least 50k records, but the app always times out
before all of the data can be downloaded. Has anyone ever run into this
problem and found a solution? I don't know if there is a way to download
smaller section of the database at a time and add it all together.
Thanks,
Bill
--
View this message in context: http://apache-flex-users.2333346.n4.nabble.com/Downloading-Large-Amounts-of-Data-from-Database-tp13415.html
Sent from the Apache Flex Users mailing list archive at Nabble.com.
Re: Downloading Large Amounts of Data from Database
Posted by kamcknig <ka...@gmail.com>.
I'm not sure how you are connecting to your DB, normally in the past I've
used PHP. So I connect to a PHP scriipt that I can send GET parameters to
and have the PHP handle the querying.
--
View this message in context: http://apache-flex-users.2333346.n4.nabble.com/Downloading-Large-Amounts-of-Data-from-Database-tp13415p13427.html
Sent from the Apache Flex Users mailing list archive at Nabble.com.
Re: Downloading Large Amounts of Data from Database
Posted by Scott <sc...@matheson.it>.
Compression is just standard web server configuration, I pull a lot of sound and enabled compression and it made a big difference
Sent from my iPad
> On 27 Aug 2016, at 21:51, bilbosax <wa...@comcast.net> wrote:
>
> Thanks for the tip! I have never heard of http compression before with
> databases. Is that something that you enable on the server side, or
> something that you set up when creating services in Flex?
>
>
>
> --
> View this message in context: http://apache-flex-users.2333346.n4.nabble.com/Downloading-Large-Amounts-of-Data-from-Database-tp13415p13442.html
> Sent from the Apache Flex Users mailing list archive at Nabble.com.
Re: Downloading Large Amounts of Data from Database
Posted by scott matheson <sc...@matheson.it>.
sorry i only know about web apps, the browse does all the work
it seems you need to write your own applications server logic, or MySQL stored proc
see this http://www.mysqltutorial.org/mysql-cursor/ <http://www.mysqltutorial.org/mysql-cursor/>
1, pass down data in chunks (my preferred approach)
2, comprises your data, there are may tools to do this, i would loom for a ZIP tool, these tools or create you own coding ie 10,000 could be 1~04 ie 1 + 4 zeros, it all depends on you data, replacing pattens with codes can save download time
> On 28 Aug 2016, at 02:11, bilbosax <wa...@comcast.net> wrote:
>
> If I compress the datastream coming from my database, do I have to somehow
> decompress it in my AIR application, or does that happen automatically?
>
>
>
> --
> View this message in context: http://apache-flex-users.2333346.n4.nabble.com/Downloading-Large-Amounts-of-Data-from-Database-tp13415p13445.html
> Sent from the Apache Flex Users mailing list archive at Nabble.com.
Re: Downloading Large Amounts of Data from Database
Posted by Alex Harui <ah...@adobe.com>.
On 8/27/16, 6:11 PM, "bilbosax" <wa...@comcast.net> wrote:
>If I compress the datastream coming from my database, do I have to somehow
>decompress it in my AIR application, or does that happen automatically?
Have you proven it is the download that is the problem as opposed to
deserialization? Otherwise you are trying to solve the wrong problem.
AIUI, server data is delivered asynchronously and does not take up the
main thread while the bytes are coming down over the network. Then, once
all of the bytes have arrived, there might be a conversion of the bits to
XML which should be in C code and pretty fast., or for RemoteObject, there
will be a deserialization/conversion to the classes. Then an
Event.COMPLETE event is fired and for non-RemoteObjects, Flex apps
generally do other deserialization then. It is this latter
deserialization that was the culprit in a few apps I've had to
troubleshoot. I've never seen the delivery of the bytes over the network
be the cause of a hung player. Compression of the bytes can reduce
bandwidth and thus the delay before the users gets to see the data, but I
don't think it prevent ActionScript from running.
My 2 cents,
-Alex
Re: Downloading Large Amounts of Data from Database
Posted by Deepak MS <me...@gmail.com>.
I'm not sure if this is what you are looking for, but here it is:
We use sqlite db for offline ipad apps. We zip the db file, place it on
server and download it from there. Zip File size ranges from 300 to 400MB.
We download the file in chunks and write it on the device and then unzip it
using ANE (as Flex/AIR unzipping takes longer time). We then query the db
and display data in the app.
Reference link:
http://deepflex.blogspot.in/2013/03/writing-large-files-on-mobileandroidios.html?m=1
On 28 Aug 2016 6:44 am, "bilbosax" <wa...@comcast.net> wrote:
> If I compress the datastream coming from my database, do I have to somehow
> decompress it in my AIR application, or does that happen automatically?
>
>
>
> --
> View this message in context: http://apache-flex-users.
> 2333346.n4.nabble.com/Downloading-Large-Amounts-of-Data-from-Database-
> tp13415p13445.html
> Sent from the Apache Flex Users mailing list archive at Nabble.com.
>
Re: Downloading Large Amounts of Data from Database
Posted by bilbosax <wa...@comcast.net>.
If I compress the datastream coming from my database, do I have to somehow
decompress it in my AIR application, or does that happen automatically?
--
View this message in context: http://apache-flex-users.2333346.n4.nabble.com/Downloading-Large-Amounts-of-Data-from-Database-tp13415p13445.html
Sent from the Apache Flex Users mailing list archive at Nabble.com.
Re: Downloading Large Amounts of Data from Database
Posted by Scott <sc...@matheson.it>.
PS you can create a node web server in a few lines of code this will give you good control on the round trip
Sent from my iPad
> On 27 Aug 2016, at 21:51, bilbosax <wa...@comcast.net> wrote:
>
> Thanks for the tip! I have never heard of http compression before with
> databases. Is that something that you enable on the server side, or
> something that you set up when creating services in Flex?
>
>
>
> --
> View this message in context: http://apache-flex-users.2333346.n4.nabble.com/Downloading-Large-Amounts-of-Data-from-Database-tp13415p13442.html
> Sent from the Apache Flex Users mailing list archive at Nabble.com.
Re: Downloading Large Amounts of Data from Database
Posted by bilbosax <wa...@comcast.net>.
Thanks for the tip! I have never heard of http compression before with
databases. Is that something that you enable on the server side, or
something that you set up when creating services in Flex?
--
View this message in context: http://apache-flex-users.2333346.n4.nabble.com/Downloading-Large-Amounts-of-Data-from-Database-tp13415p13442.html
Sent from the Apache Flex Users mailing list archive at Nabble.com.
Re: Downloading Large Amounts of Data from Database
Posted by Scott Matheson <sc...@matheson.it>.
Just checking have you enabled HTTP compressional it makes a big difference
The standard approach is a SQlL causer, this is one SQL quire the server will pass you say 10000 records, the client then ask is for the next and so on
Or you could do the same by limiting the data retuned you would write some server side code to pass 10000 records at a time
Sent from my iPhone
> On 27 Aug 2016, at 03:22, bilbosax <wa...@comcast.net> wrote:
>
> That was my thinking as well, I just need a little direction. I know how to
> use SQL in Flex when connecting to a local SQLite database. But the only way
> I have connected to a MySQL database is by using data services and you are
> only given a few options like GetAll and Get_paged and count(). Is there a
> way to send more specific SQL commands to a MySQL database than the preset
> data service commands?
>
>
>
> --
> View this message in context: http://apache-flex-users.2333346.n4.nabble.com/Downloading-Large-Amounts-of-Data-from-Database-tp13415p13421.html
> Sent from the Apache Flex Users mailing list archive at Nabble.com.
Re: Downloading Large Amounts of Data from Database
Posted by bilbosax <wa...@comcast.net>.
That was my thinking as well, I just need a little direction. I know how to
use SQL in Flex when connecting to a local SQLite database. But the only way
I have connected to a MySQL database is by using data services and you are
only given a few options like GetAll and Get_paged and count(). Is there a
way to send more specific SQL commands to a MySQL database than the preset
data service commands?
--
View this message in context: http://apache-flex-users.2333346.n4.nabble.com/Downloading-Large-Amounts-of-Data-from-Database-tp13415p13421.html
Sent from the Apache Flex Users mailing list archive at Nabble.com.
Re: Downloading Large Amounts of Data from Database
Posted by Kyle McKnight <ka...@gmail.com>.
If you know the number of records, maybe a separate query. One query to
get the number of records, you could then pull x number of records at a
time and keep track of how many you have downloaded versus how many you
need to download and then send alone the start and limit of the query so
that you can get the next set of data. Then as each set comes in add it to
your "global" array/collection.
Kyle McKnight
Senior UI Engineer - Accesso
602.515.1444 (M)
On Fri, Aug 26, 2016 at 9:42 PM, bilbosax <wa...@comcast.net> wrote:
> Yes, unfortunately I do need all of the records at once. My application
> is a
> huge sorting and filtering machine and it only works when it has all of the
> records. I looked at paging and wondered if there was a way to use it
> without it being tied to the scrolling of a grid. If I could download a
> thousand records at a time and just loop through it 50 times to get all of
> the data, that would be ideal. I don't know if paging can be requested in
> groups, or if it only works when tied to a grid component. Anybody know?
>
>
>
> --
> View this message in context: http://apache-flex-users.
> 2333346.n4.nabble.com/Downloading-Large-Amounts-of-Data-from-Database-
> tp13415p13419.html
> Sent from the Apache Flex Users mailing list archive at Nabble.com.
>
Re: Downloading Large Amounts of Data from Database
Posted by Alex Harui <ah...@adobe.com>.
On 8/26/16, 6:42 PM, "bilbosax" <wa...@comcast.net> wrote:
>Yes, unfortunately I do need all of the records at once. My application
>is a
>huge sorting and filtering machine and it only works when it has all of
>the
>records. I looked at paging and wondered if there was a way to use it
>without it being tied to the scrolling of a grid. If I could download a
>thousand records at a time and just loop through it 50 times to get all of
>the data, that would be ideal. I don't know if paging can be requested in
>groups, or if it only works when tied to a grid component. Anybody know?
There are several possible solutions. But first, again, profiling
information would be a good first step. It may not be that the fetch of
the data is taking too long, but rather, the deserialization of the data
into ActionScript objects. Deserialization could be done in a Worker
(there's that word again!).
You could also fetch chunks of data by chaining requests without the need
for workers, or let the scrolling dictate what chunk of data to get.
A common trick is to actually pop up a progress bar so it tracks your 50
requests before letting folks interact with the DataGrid.
-Alex
Re: Downloading Large Amounts of Data from Database
Posted by bilbosax <wa...@comcast.net>.
Yes, unfortunately I do need all of the records at once. My application is a
huge sorting and filtering machine and it only works when it has all of the
records. I looked at paging and wondered if there was a way to use it
without it being tied to the scrolling of a grid. If I could download a
thousand records at a time and just loop through it 50 times to get all of
the data, that would be ideal. I don't know if paging can be requested in
groups, or if it only works when tied to a grid component. Anybody know?
--
View this message in context: http://apache-flex-users.2333346.n4.nabble.com/Downloading-Large-Amounts-of-Data-from-Database-tp13415p13419.html
Sent from the Apache Flex Users mailing list archive at Nabble.com.
Re: Downloading Large Amounts of Data from Database
Posted by Justin Mclean <ju...@classsoftware.com>.
Hi,
> I don't know if there is a way to download smaller section of the database at a time and add it all together.
Do you need all of the data at once? If not then one solution is that you can page the data and download only what the user need and a little either side of that so that it’s responsive. AMF supports this [1].
If you do need all the dat at once, do you need all of the columns? Try to cut down on the columns and/or break up the query to return X rows at a time and make multiple requests. In mySQL you can do this with LIMIT n,n at the end of the SQL.
Thanks,
Justin
1. http://www.adobe.com/devnet/flash-builder/articles/flashbuilder_datapaging_php.html
Re: Downloading Large Amounts of Data from Database
Posted by Daniel Webster <dw...@yahoo.com.au.INVALID>.
Hi Bill,
Not sure how much this will help but you can reduce the size of the meta data or variable names being transferred.On the server side when preparing the data to return to the client (or via SQL aliases), you can change all of the variable names in each row to something very short instead (e.g. "first_name" becomes "fn", "comment" becomes "cm" etc). This can really reduce the amount of data returned, especially if there are a lot of columns.Or, return each row as an indexed array only so there are no variable names at all to transfer along with the data.In the client you will then have to map them back to the original names.
I have a habit of returning the same database column names as the variable names but I try to rename/alias them if I have time.
Here's an example in json:
[ {"id": 1, "person_name": "Jim", "address": "123 test road"}, {"id": 2, "person_name": "Phil", "address": "63 test road"}]
(141 bytes)
now becomes
[ {"id": 1, "pn": "Jim", "ad": "123 test road"}, {"id": 2, "pn": "Phil", "ad": "63 test road"}](111 bytes ~ 22% reduction)
Daniel
From: bilbosax <wa...@comcast.net>
To: users@flex.apache.org
Sent: Saturday, 27 August 2016, 10:50
Subject: Downloading Large Amounts of Data from Database
I am running into an issue that I believe is going to take some creative
problem solving. I have created a service to download large amounts of data
from a MySQL database, at least 50k records, but the app always times out
before all of the data can be downloaded. Has anyone ever run into this
problem and found a solution? I don't know if there is a way to download
smaller section of the database at a time and add it all together.
Thanks,
Bill
--
View this message in context: http://apache-flex-users.2333346.n4.nabble.com/Downloading-Large-Amounts-of-Data-from-Database-tp13415.html
Sent from the Apache Flex Users mailing list archive at Nabble.com.