Log in to AnswerLog in to Follow
Somesh Renganathan
33Views
7Comments
[Data Grid Reactive] Processing 2 Million+ records on Screen
Question
Forge
Data Grid Reactive
Forge component by Bruno Martinho

We have started using the forge component in our project and it is working find for smaller records. But when it reaches  more than 100k+ records, the list to object and the object to json takes time and the grid does not even populate the records on the screen. We tried several approaches, but the grid does not load data or the connection timeout happens. The table is huge with around million records and 50 columns. Is there any way to load the million records onto screen within 2-3 seconds?

0
0
on 11th Feb

My understanding is that the component is still in development, so hopefully there is a solution for this.  For now though, I do not think there is.  This data grid (as well as the advanced data grid for traditional web) loads all the data into the grid in its initial state.  I'm impressed that it was able to load 100k records for you, because that is actually quite a bit, considering they all need to be in memory.

Personally, I think the best solution for this is the ability to do server-side filtering in the columns.  This way, column filtering (and the other powerful features of the component) could still be used while not needing to load all the data into memory at the beginning.

0
0
on 11th Feb

Hello Somesh,

Currently the data grid need to receive all data to the client to be able to execute the functions like filter, sort and group without round trips to the server.

We are considering to add the ability to do some of these operations server side.

Can you please let us know what kind of functionalities you need in the grid, if you have the possibility to use server side? Is is pagination, filtering, sorting? What about grouping?

Thanks,

Bruno Martinho

3
0
on 11th Feb

Hi Bruno,

Yes, since there is a process which gets the list items and converting it into objects and then back to JSON and feeding it to the grid input, guess the performance varies as per the end user machine processing power and RAM, which I believe needs to be done at the server and need only the data to be shown. 

Currently, we believe it would be good if we could see some initial data (say first 50k records) and then in background it can fetch the remaining data. Pagination needs to be optimized to show atleast some data, instead of loading message. All other features are amazing. If it can handle huge data, it would be the perfect component to process data on the web application.

Thanks,

Somesh

0
0
on 12th Feb

Replying to Somesh Renganathan's comment on 12 Feb 2021 06:40:32

Hello Somesh,

We are currently investigating a way to allow interaction between the data grid and the data source and allow to do some hard processing on the server side.

I'll keep you posted when we have progress on that and even ask a few more follow up question related to the use case.

Thanks,

Bruno

1
0
on 15th Feb
Tom Zhao
Champion

Hi

Sorry, If I misunderstanding your issue.

First, I don't think loading Million+ records on a Screen is a good idea. What is the use case for this? 

Second, What is the original data source in this case? If you can control it(Access, Filter etc), You can expose an API. In the API to access the original datasource and fetch the smaller recordset you needed, then return it to the screen.

Furthermore, If possible you can implements API that supports pagination such as(e.g: top, skip )

Kind Regards

2
0
on 12th Feb

Hi Somesh,

This is not the right approach. Showing these many records in Datagrid would be a huge performance issue. Dont show more than 500 records in  a page

1. Show Pagination and the pagination should be server side . So for an example you are showing 100 records in a page than only 100 records should be send to Client Side (Sending all the record to client side would hang the browser)

2.Don't  fetch all the records in server side only fetch a single page result from DB. Keeping millions of item in server memory (RAM) would cause huge issue in high concurrency.

Best Regards

Devendra BAghel





0
0
on 12th Feb

@Bruno Martinho is there any update on this?
Per business requirement I need to show around 1k records but I get a "connection has timed out" error, even though the data action finishes a bit later from getting the data. Will we be able to set some wait till IsDataFetched?

Best regards,
Miguel Garcia

0
0
on 8th Apr
Community GuidelinesBe kind and respectful, give credit to the original source of content, and search for duplicates before posting.