Connection to a huge database CSV

Not so long ago in FastReport .NET was added the ability to load data from a csv file. In other words, a CSV file can now be used as a data source.

This was done after some time from one of the users entered an interesting query to a support. He claimed that he has a CSV file that does not open, instead, an error message appears. The user kindly provided this file for testing.

At first glance, it was thought that the file too is big, that is the reason why it can not be opened. But upon further examination it was revealed that the problem is not in the size of the file. As it turned out, the file does not entirely correspond to the CSV specification. In the file rows were found that did not contain separators. Apparently, they are used as comments, with any data yourself.

It was decided to ignore such lines when data is loaded. As a result file has become loaded without any problems.

It would seem that the problem is solved and we can settle down, but the idea of a large csv file did not give a rest. After some searching there was found a really huge file. This file size was 441 MB, seemingly not very awesome, but, nevertheless, it consisted of 31 columns and 2 458 525 lines.

Testing this download brought dual results. On the one hand, the file opens that undoubtedly pleased. But it was a very sad fact that the file download took nearly half an hour.

It is known two solutions to this problem. However, buying a new computer is not a good idea. Therefore, I had to spend some time to optimize the code. As a result, the file download has been achieved within two minutes