Fill a C# DataTable "on disk" rather than "in memory" to avoid Memory overflow

I need to fill two DataTable from two different databases for some comparisons and read-only operations. That takes me to "select *" into these two DataTable.

However, the table I need to retrieve contains about 3 million rows in both databases. I get an OutOfMemoryExceptionwhile filling.

Is there a possibility to have these DataTable on disk rather than in memory? Do you see any other solution to compare/perform read-only on such two huge sets? The things that I need to do make it very complicated to proceed by batches of subsets of rows.

Answers


My guess is that the ideal/efficient way of performing this would be to do it within the database. Unfortunately, your two tables are in two separate databases, which may cause complications.

Are these tables in two different databases on the same server (as in SQL Server instance), or are they in two separate servers? If they are within the same server/instance, then you could probably perform your comparison logic on the database server.

Otherwise, if these tables are on separate servers, then is there any possibility of setting up some kind of replication between databases so that you can get both data sets into a single database where you can perform your comparisons?


What about sorting both of them by key.

i.e.

set1    A  A  A  B  C            E
                 |            
set2             B       D

then you can go on set1, change to set2, go back to set1... makes iterating much easier.


Need Your Help

PHP session seemingly not working

php session cookies session-variables

Session variables are apparently not working for me. I don't know what I'm doing wrong. This is the code that I'm using to check for a valid password:

How to disable additivity on marker logger?

java logging configuration log4j log4j2

I have defined a Marker and would like to log this only to a specific file. Therefore I'm trying to set additivity = false. But it does not work and is still also logged to my global logfile in add...