JavaScript Grid with One Million Records

Some time ago, I got a comment from a user who wanted to use the grid as a log table. He quickly put 10,000 records into the grid without paginating, but found that it got really, really slow. This got me wondering, can it be optimized to handle 10k of records or it is too much for the browser.

Establishing the Baseline

For my initial test I have used w2ui ver 1.2, created a simple grid with 4 columns, populated it with random data and rendered into a 1024x768 container. I have started my testing with recordsPerPage set to 50 and tried 25, 250, 2.5K, 25K, 250K and 1MIL records in the grid. To my surprise the grid performed well. It was responsive and fast, I could go from page to page and perform other grid functions. It was an unexpected and pleasant surprise. It meant that JavaScript can handle large data sets, which was awesome. As a next step, I have set recordsPerPage to 1,000,000 and repeated my tests. The table below shows my findings:

# of Records	Render Time	# of DOM Nodes
25	0.047 sec	329
250	0.466 sec	2,354
2,500	14.425 sec	22,604
25,000	crashed	??
250,000	crashed	??
1,000,000	crashed	??

I have found that if the number of records in the grid becomes more then just a few thousands the grid gets very slow because the rendering speed is directly related to the number of nodes in the DOM. If the number of nodes in the DOM is more then 40-50k (depending on your computer configuration and amount of memory), your browser will crash or will become unresponsive. So, I decided to set out on a quest to do two things: (1) dynamically create records as user scrolls (2) optimize grid to handle large data sets. After a few weeks of work, the grid was optimized and ready for testing. I have repeated my original tests:

# of Records	Render Time	# of DOM Nodes
25	0.021 sec	355
250	0.112 sec	2,605
2,500	0.036 sec	705
25,000	0.051 sec	705
250,000	0.198 sec	705
1,000,000	0.676 sec	705

Wow! The results were awesome! Now the grid was capable of handling large data sets that not so long ago only a database server could. If you are wondering why 2500 records performed better than 250, the answer is very simple: I have defined a parameter in the grid not to use buffered scroll if number of records below 300. So, when there are less then 300 records - all of them are rendered in the grid. If there are more then 300 records, only the ones that are in the view, plus a few items on top and bottom for smooth scrolling.

Example of the grid

Below you can find an example of the grid with random 25K records. You can generate a different amount and play with it for yourself. Please note that different browsers can handle different number of records, though pretty much all of them can handle 1MIL of records. Generate:

Sorting & Searching

The next challenge was to make local sorting and searching fast. In the table below you can see results before optimization:

# of Records	Sorting (int)	Sorting (Text)	Searching (int)	Searching (text)
25	0.000 sec	0.000 sec	0.005 sec	0.005 sec
250	0.002 sec	0.002 sec	0.029 sec	0.029 sec
2,500	0.022 sec	0.010 sec	0.239 sec	0.245 sec
25,000	0.243 sec	0.066 sec	2.539 sec	2.282 sec
250,000	3.046 sec	0.632 sec	25.725 sec	25.204 sec
1,000,000	14.804 sec	2.353 sec	??	??

As it is seen from the data, the sorting performed well (I used Array.sort() for this) and the search was increasingly slow with the increase in the number of records. I found that the biggest issue with the search was the use of eval() to parse nested record sets, which turned out to be slow. I have re-factored the code and repeated the tests

# of Records	Sorting (int)	Sorting (Text)	Searching (int)	Searching (text)
25	0.000 sec	0.000 sec	0.000 sec	0.000 sec
250	0.001 sec	0.001 sec	0.001 sec	0.001 sec
2,500	0.019 sec	0.006 sec	0.006 sec	0.008 sec
25,000	0.275 sec	0.069 sec	0.058 sec	0.052 sec
250,000	3.622 sec	0.702 sec	0.589 sec	0.523 sec
1,000,000	16.301 sec	2.789 sec	2.456 sec	2.019 sec

Infinite Scroll

Now, rendering, sorting and searching was fast on large data sets, but still was not good enough if the number of records goes over 250K. I did not know any other way to make it faster and I think it hits the performance ceiling just because there is nothing else you can do to optimize it. There is no way to create indexes in JavaScript, but database already have this functionality. So, to make your grid work on large data set you can use Infinite Scroll, which I have also implemented in the grid. I have created a Postgres database with 1 million records and implemented infinite scroll in the grid, buffering 100 records at a time. All my test have shown that it was never over 0.2 seconds to sort and search through this record set.

Browsers Test

Initially, I have done all my tests in Chrome. Now, I wanted to repeat them on other browsers too. Though Chrome performed better in most categories, the test have shown that other browsers can do a pretty good job with large data sets. I have recorded best of 5 tries for each browser in each category. Render - (green - best, yellow - second best)

# of Records	Chrome	FireFox	Safari 6	Opera	IE 9
25	0.028 sec	0.047 sec	0.022 sec	0.037 sec	0.044 sec
250	0.153 sec	0.251 sec	0.131 sec	0.197 sec	0.401 sec
2,500	0.043 sec	0.068 sec	0.034 sec	0.055 sec	0.072 sec
25,000	0.058 sec	0.075 sec	0.080 sec	0.077 sec	0.088 sec
250,000	0.220 sec	0.116 sec	0.124 sec	0.270 sec	0.252 sec
1,000,000	0.734 sec	0.254 sec	0.393 sec	1.002 sec	0.797 sec

Sort - (green - best, yellow - second best)

# of Records	Chrome	FireFox	Safari 6	Opera	IE 9
25	0.000 sec	0.000 sec	0.000 sec	0.000 sec	0.000 sec
250	0.001 sec	0.004 sec	0.004 sec	0.002 sec	0.002 sec
2,500	0.013 sec	0.023 sec	0.028 sec	0.014 sec	0.007 sec
25,000	0.071 sec	0.197 sec	0.358 sec	0.134 sec	0.106 sec
250,000	0.754 sec	2.176 sec	5.134 sec	1.421 sec	1.719 sec
1,000,000	2.907 sec	9.347 sec	24.038 sec	6.040 sec	7.709 sec

Search - (green - best, yellow - second best)

# of Records	Chrome	FireFox	Safari 6	Opera	IE 9
25	0.000 sec	0.001 sec	0.000 sec	0.000 sec	0.000 sec
250	0.001 sec	0.001 sec	0.001 sec	0.001 sec	0.001 sec
2,500	0.008 sec	0.013 sec	0.009 sec	0.012 sec	0.005 sec
25,000	0.082 sec	0.050 sec	0.089 sec	0.114 sec	0.070 sec
250,000	0.807 sec	1.728 sec	1.225 sec	0.932 sec	1.114 sec
1,000,000	3.332 sec	7.674 sec	4.903 sec	3.581 sec	5.079 sec

The table below shows a winner, where I have given a browser a point if it were the best in the category and half a point if it were second best:

Chrome	9.5
Safari 6	4.0
FireFox	3.5
IE 9	3.5
Opera	2.5

Browser Limits

I have discovered an interested browser limitation - the height of the div has a limit (so is scrollable area in the div). Hence, number of scrollable records has a limit since one record is 25px. I have created a simple page with one div and by trial and error I have discovered the max height of the div:

Browser	Height Limit	Record Limit
Opera 12	1,677,720,027,136 px	67,108,801,085
Safari 6	1,677,720,518 px	66,708,820
Chrome 28	33,554,420 px	1,342,176
FireFox 22	17,895,697 px	715,827
IE 9	10,739,975 px	429,599

Different browsers behave differently when you try to create a div with height over the limit. Opera will simply max it up to the limit and ignore anything over it. FireFox will set the height of the div to 0 if height it too big. Safari, will display distorted view if it is too large and Chrome will display a black box at the bottom. I have also discovered that Chrome has a smaller limit of height for the body. For the body, height can be no more then 16,777,201 px.

Important Lessons

During this exercise I have learned several important things:

• Large number of DOM nodes make rendering slow
• JavaScript arrays can handle large data sets
• Looping through large arrays is fast
• Sorting arrays by providing custom function to Array.sort() is fast
• eval() is slow, should not be used in large loops
• To achieve smooth scrolling render a few hidden records on top and bottom outside of the visible area

I think that 1MIL of records for JavaScript is too much, though it is doable. If user has to wait over a second it makes user experience sluggish and unpleasant. But as seen in the tables above any browser can give you a good user experience with 100K of records or less.

User Comments