Some time ago, I got a comment from a user who wanted to use the grid as a log table. He quickly put
10,000 records into the grid without paginating, but found that it got really, really slow. This got me wondering,
can it be optimized to handle 10k of records or it is too much for the browser.
Establishing the Baseline
For my initial test I have used w2ui ver 1.2, created a simple grid with 4 columns, populated it with random data and rendered
into a 1024x768 container. I have started my testing with recordsPerPage set to 50 and tried 25, 250, 2.5K, 25K, 250K and
1MIL records in the grid. To my surprise the grid performed well. It was responsive and fast, I could go from page to page
and perform other grid functions.
It was an unexpected and pleasant surprise. It meant that JavaScript can handle large data sets, which was awesome. As a
next step, I have set recordsPerPage to 1,000,000 and repeated my tests. The table below shows my findings:
# of Records
Render Time
# of DOM Nodes
25
0.047 sec
329
250
0.466 sec
2,354
2,500
14.425 sec
22,604
25,000
crashed
??
250,000
crashed
??
1,000,000
crashed
??
I have found that if the number of records in the grid becomes more then just a few thousands the grid gets very slow because the
rendering speed is directly related to the number of nodes in the DOM. If the number of nodes in the DOM is more then 40-50k
(depending on your computer configuration and amount of memory), your browser will crash or will become unresponsive.
So, I decided to set out on a quest to do two things: (1) dynamically create records as user
scrolls (2) optimize grid to handle large data sets. After a few weeks of work, the grid was optimized and
ready for testing. I have repeated my original tests:
# of Records
Render Time
# of DOM Nodes
25
0.021 sec
355
250
0.112 sec
2,605
2,500
0.036 sec
705
25,000
0.051 sec
705
250,000
0.198 sec
705
1,000,000
0.676 sec
705
Wow! The results were awesome! Now the grid was capable of handling large data sets that not so long ago only a database server could.
If you are wondering why 2500 records performed better than 250, the answer is very simple: I have defined a parameter in the
grid not to use buffered scroll if number of records below 300. So, when there are less then 300 records - all of them
are rendered in the grid. If there are more then 300 records, only the ones that are in the view, plus a few items on top and
bottom for smooth scrolling.
Example of the grid
Below you can find an example of the grid with random 25K records. You can generate a different amount and play with it for
yourself. Please note that different browsers can handle different number of records, though pretty much all of them can handle 1MIL
of records.
Generate:
Sorting & Searching
The next challenge was to make local sorting and searching fast. In the table below you can see results before
optimization:
# of Records
Sorting (int)
Sorting (Text)
Searching (int)
Searching (text)
25
0.000 sec
0.000 sec
0.005 sec
0.005 sec
250
0.002 sec
0.002 sec
0.029 sec
0.029 sec
2,500
0.022 sec
0.010 sec
0.239 sec
0.245 sec
25,000
0.243 sec
0.066 sec
2.539 sec
2.282 sec
250,000
3.046 sec
0.632 sec
25.725 sec
25.204 sec
1,000,000
14.804 sec
2.353 sec
??
??
As it is seen from the data, the sorting performed well (I used Array.sort() for this) and the search was increasingly slow with the
increase in the number of records. I found that the biggest issue with the search was the use of eval() to parse nested record sets,
which turned out to be slow. I have re-factored the code and repeated the tests
# of Records
Sorting (int)
Sorting (Text)
Searching (int)
Searching (text)
25
0.000 sec
0.000 sec
0.000 sec
0.000 sec
250
0.001 sec
0.001 sec
0.001 sec
0.001 sec
2,500
0.019 sec
0.006 sec
0.006 sec
0.008 sec
25,000
0.275 sec
0.069 sec
0.058 sec
0.052 sec
250,000
3.622 sec
0.702 sec
0.589 sec
0.523 sec
1,000,000
16.301 sec
2.789 sec
2.456 sec
2.019 sec
Infinite Scroll
Now, rendering, sorting and searching was fast on large data sets, but still was not good enough if the number of records goes
over 250K. I did not know any other way to make it faster and I think it hits the performance ceiling just because there is
nothing else you can do to optimize it. There is no way to create indexes in JavaScript, but database already have this functionality.
So, to make your grid work on large data set you can use Infinite Scroll, which I have also implemented in the grid.
I have created a Postgres database with 1 million records and implemented infinite scroll in the grid, buffering
100 records at a time. All my test have shown that it was never over 0.2 seconds to sort and search through this record set.
Browsers Test
Initially, I have done all my tests in Chrome. Now, I wanted to repeat them on other browsers too.
Though Chrome performed better in most categories, the test have shown that other
browsers can do a pretty good job with large data sets. I have recorded best of 5 tries for each browser in each category.
Render - (green - best, yellow - second best)
# of Records
Chrome
FireFox
Safari 6
Opera
IE 9
25
0.028 sec
0.047 sec
0.022 sec
0.037 sec
0.044 sec
250
0.153 sec
0.251 sec
0.131 sec
0.197 sec
0.401 sec
2,500
0.043 sec
0.068 sec
0.034 sec
0.055 sec
0.072 sec
25,000
0.058 sec
0.075 sec
0.080 sec
0.077 sec
0.088 sec
250,000
0.220 sec
0.116 sec
0.124 sec
0.270 sec
0.252 sec
1,000,000
0.734 sec
0.254 sec
0.393 sec
1.002 sec
0.797 sec
Sort - (green - best, yellow - second best)
# of Records
Chrome
FireFox
Safari 6
Opera
IE 9
25
0.000 sec
0.000 sec
0.000 sec
0.000 sec
0.000 sec
250
0.001 sec
0.004 sec
0.004 sec
0.002 sec
0.002 sec
2,500
0.013 sec
0.023 sec
0.028 sec
0.014 sec
0.007 sec
25,000
0.071 sec
0.197 sec
0.358 sec
0.134 sec
0.106 sec
250,000
0.754 sec
2.176 sec
5.134 sec
1.421 sec
1.719 sec
1,000,000
2.907 sec
9.347 sec
24.038 sec
6.040 sec
7.709 sec
Search - (green - best, yellow - second best)
# of Records
Chrome
FireFox
Safari 6
Opera
IE 9
25
0.000 sec
0.001 sec
0.000 sec
0.000 sec
0.000 sec
250
0.001 sec
0.001 sec
0.001 sec
0.001 sec
0.001 sec
2,500
0.008 sec
0.013 sec
0.009 sec
0.012 sec
0.005 sec
25,000
0.082 sec
0.050 sec
0.089 sec
0.114 sec
0.070 sec
250,000
0.807 sec
1.728 sec
1.225 sec
0.932 sec
1.114 sec
1,000,000
3.332 sec
7.674 sec
4.903 sec
3.581 sec
5.079 sec
The table below shows a winner, where I have given a browser a point if it were the best in the category and half a point if it
were second best:
Chrome
9.5
Safari 6
4.0
FireFox
3.5
IE 9
3.5
Opera
2.5
Browser Limits
I have discovered an interested browser limitation - the height of the div has a limit (so is scrollable area in the div).
Hence, number of scrollable records has a limit since one record is 25px. I have created a simple page with one div and by trial
and error I have discovered the max height of the div:
Browser
Height Limit
Record Limit
Opera 12
1,677,720,027,136 px
67,108,801,085
Safari 6
1,677,720,518 px
66,708,820
Chrome 28
33,554,420 px
1,342,176
FireFox 22
17,895,697 px
715,827
IE 9
10,739,975 px
429,599
Different browsers behave differently when you try to create a div with height over the limit. Opera will simply max it up to the limit
and ignore anything over it. FireFox will set the height of the div to 0 if height it too big. Safari, will display distorted
view if it is too large and Chrome will display a black box at the bottom. I have also discovered that Chrome has a smaller limit of height
for the body. For the body, height can be no more then 16,777,201 px.
Important Lessons
During this exercise I have learned several important things:
Large number of DOM nodes make rendering slow
JavaScript arrays can handle large data sets
Looping through large arrays is fast
Sorting arrays by providing custom function to Array.sort() is fast
eval() is slow, should not be used in large loops
To achieve smooth scrolling render a few hidden records on top and bottom outside of the visible area
I think that 1MIL of records for JavaScript is too much, though it is doable. If user has to wait over a second it makes
user experience sluggish and unpleasant. But as seen in the tables above any browser can give you a good user experience with
100K of records or less.