Moving large dataset across django databases

What's the best way to move large dataset from one django database to another?

I would like to do this at high level (in django) and not at the database level. All the existing tools that I know of (dumpdata/loaddata, serializer class, django-extensions) are in memory so it doesn't handle large datasets.

Answers


I found this post very useful - Migrating Django from MySQL to PostgreSQL the Easy Way.


I had the same problem with several apps/databases, so wrote this script, which is a fork of django's dumpdata, but dumps data in chunks to avoid MemoryError

Script is available at https://github.com/fastinetserver/django-dumpdata-chunks

Example usage:

1) Dump data into many files:

mkdir some-folder

./manage.py dumpdata_chunks your-app-name
--output-folder=./some-folder --max-records-per-chunk=100000

2) Load data from the folder:

find ./some-folder | egrep -o "([0-9]+_[0-9]+)" | xargs ./manage.py loaddata

Need Your Help

Removing a subset of a dict from within a list

python list containers

This is really only easy to explain with an example, so to remove the intersection of a list from within a dict I usually do something like this:

How can I make this VBA loop (for each cell, copy paste) faster?

excel vba excel-vba

I have a piece of code that is taking up a large amount of the actual runtime. It seems like this loop actually makes Excel unresponsive at times (not 100% sure about this, but this seems to me the...