COVID-19 When the report is too small for your data

Do you store COVID-19 data in Excel?

Can the report show not all data?

Here - https://uk.news.yahoo.com/missing-coronavirus-tests-glitch-large-excel-spreadsheet-file-095054235.html - you can read a wonderful story about how almost 16,000 cases of Covid-19 infection were left out in the UK because the Excel spreadsheet ran out of space.

If you are lazy to google, let me remind you that the limit of an Excel spreadsheet is 65,000 lines.

These are not isolated cases! People have no idea how much data they and their customers will have to deal with, and what are the capabilities of the tools they use!

And then such articles appear: https://www.techrepublic.com/blog/microsoft-office/10-things-you-should-never-do-in-excel/

So what can we do?

1. Use databases to store data!

It might sound completely crazy, but always and everywhere, when you decide to create a list, a base (it’s my professional one – I always use this term), a data storage system – pick up a database. Estimate the amount of data. Your descendants will be grateful for that!

Our example – in our CRM (by the way, our CRM is one of the first online ones – we didn’t even know this term) there are hundreds of thousands of registrations, clients, records, and partner registrations. It works!

Even now, when dozens of people from all over the world work with it at the same time, I cannot call it a “high-loaded system”, because we know how high-loaded systems really work and what big data actually is.

2. Check if your reporting system copes with this amount of data.

Interesting clients have often come to us, especially recently. Contractors who have handed over their projects to the customers for half a year already need to change the reporting system to ours in already implemented and working projects.

They took the project seriously, considered the toolkit, components, compared them according to several criteria (perhaps they did not even consider our system, or they did, but for some reason FastReport did not make it into the final), but then the project was filled with real data, worked with it and… fell. (At best, “displaying first 65,000 records”, at worst – “eating up all the allocated memory and showing nothing”).

I am far from thinking that modern developers do not pay enough attention to competent design of the system load.

Moreover, all other elements of the systems can cope with this load. And then the system falls exactly at the step of sampling data into the report generator!

And real data - it can be millions of records (this is only a sample, not the entire database, of course!) - from which it is often necessary to generate documents, for example, for subscribers.

For me, it was a revelation that there are quite old and eminent players in this market who cannot cope with this task. At best, the reports generation can take hours, or it simply crashes and does not reach the formation of a report at all.

And today, when I am asked "how did you manage that the banks, the Pension Fund, the Social Insurance Fund, manufacturers of high-load corporate systems, billing in real time, use your product, FastReport?" - I have no choice but to answer "actually, they had no other choice."

I do not urge to use only and exclusively FastReport ("buy our elephants!"), but I strongly advise developers to subject all components of their systems to the most severe stress testing (as we do), not to disappoint customers and not to risk their reputation.

This is especially true for billing systems, CRM, ERP - where there is a constant live work with clients.

3. Measure the informativeness of your report.

Yes, we have text examples of reports (that contain millions of records), where we evaluate the quality and speed of building the documents (the report has several tens of thousands of pages, which are then converted to PDF without the quality loss). It was a challenge to ourselves - we were not satisfied with the speed of the report generation, and we accelerated it three times.

Here is a link to an example of building a report from BigData.

Well, this is the BigData.

I will leave the pioneer’s joy of “how does it work in other report generators?” to grateful readers.

This is not bad when generating electricity bills for megacities residents, for example, but how informative is this report itself?

Therefore, highlight the main things, use charts - so that the data is not lost, but turns into the information covered by the eye. As, for example, did one of the winners of our competition for the best report - as you can see, big data about COVID-19 is rolled up into a reliable, and informative document.

FastReport products are always ready to quickly generate accurate reports using your data of any size.

Or you can just ask us – we are pretty good with all popular document formats and DBMSs, and can tell you where exactly “the deceit of any format” may await you.