Creating a highly queried table in MongoDB dynamically (from code) in Python Django or SubDocuments, which one?

I am going to create GPS Tracking Service using Django, Python and MongoDB. The vehicle tables/documents are going to be created dynamically on activation which will store the GPS data coming in from the devices via TCP connection (approximately we will keep capacity for 10000~100000 devices sending data in a minute) using Twisted Framework. So under the circumstances, I wanted to know if it is a good idea to create this document dynamically. If possible, please suggest a schema. I am proposing one here:

class Device_<id>(Document):
  time_of_data = DateTimeField()
  location = GeoPointField()
  speed = DecimalField()
  bearing = DecimalField()
  sensor = StringField()

class DeviceOwner(Document):
  user = ReferenceField(User)
  device = StringField() #This will store the name of the Device_<id>

The other option is to put all locations in one Document which I am not sure will be able to take the load or connection pool will cover, or even if it does if it the indexing works on SubDocuments or not. In such a case the design may be:

class Device(Document):
  user = ReferenceField(User)
  name = StringField()

class DeviceData(Document):
  time_of_data = DateTimeField()
  location = GeoPointField()
  speed = DecimalField()
  bearing = DecimalField()
  sensor = StringField()
  device = ReferenceField(Device)

*New Edit* on 4 Feb:

The other possible table design that I am considering is where loc is basically BSON.SON() data holding "x" and "y" values:

    device_locations = {"device":device[0]["_id"], 
      "locations":{
      "time":datetime.now(),
      "loc":loc,
      "status":"A",
      "engine_sensor":True,
      "ac_sensor":True,
      "temperature_sensor":0.0,
      "door1_sensor":False,
      "door2_sensor":False,
      "door3_sensor":False,
      "door4_sensor":False,
       },
      "active":True, 
      "created_on":datetime.datetime.now(),
      "modified_on":datetime.datetime.now(),
      "created_by":user,
      "modified_by":user
     }

The other option is to have separate collection for each device (btw, if the code below is a representation of that or not):

    device_locations = {
      "%s"%device[0]["_id"]:{
         "time":datetime.now(),
         "loc":loc,
          "status":"A",
          "engine_sensor":True,
         "ac_sensor":True,
         "temperature_sensor":0.0,
         "door1_sensor":False,
         "door2_sensor":False,
         "door3_sensor":False,
         "door4_sensor":False,
       },
      "active":True, 
      "created_on":datetime.datetime.now(),
      "modified_on":datetime.datetime.now(),
      "created_by":user,
      "modified_by":user
     }

Answers


A couple of points:

  1. Indexing does work on subdocuments and their related fields
  2. How big is each document going to grow? If you intend to add dynamic incremental data to the document itself, you have to be aware of limitations like 16MB max size for a document in MongoDB.
  3. Also, if you are updating same document from multiple threads / connections that can be conflicting with each other, you need to think a bit deeper in how to handle concurrency on your objects. MongoDB provides atomic operation on a document via "update" and conditional atomic updates via "findAndModify".
  4. What are your query patterns and have you thought on non-functional needs in terms of performance etc.?
  5. From definition you data seems to be something that may have certain lifetime, do you intend to purge your data at periodic interval (trying to see if TTL indices would be useful here)?

Need Your Help

CurrentCellDirtyStateChanged commits too soon

c# .net winforms .net-2.0

I have a DataGridView with a text column and a checkbox column. When the user clicks the checkbox I want to prompt the user. I've got this working using the code below:

Two arrays to be evaluated element by element in query

ruby-on-rails-3 postgresql activerecord rails-activerecord

Sorry for unclear title, however, I hope the body describes my problem clearly.