Long Term Data Acquisition

I am trying to run a Python script that is meant for long term lab data acquisition (~1 month). The script below is supposed to fetch data from an SQL database and fetch data from a DAQ device and save it to a text file. It then plots 1 hours worth of data in real time. The real time plotting has been commented out due to memory errors I have been getting, but this is not the issue I am concerned about.

Running the code for between 3-4 days will give me the error below:

Trackback (most recent call last):
 file *** line 105 in <module>
    deltay=float(float(tupline[3]/10-float(tupline2[3])/10)
TypeError: 'NoneType' object has to attribute '__getitem__'

Any thoughts on how I can get this script to run for a longer period of time?

import ctypes
from UniversalLibrary import *
import time
import datetime
from numpy import *
from scipy import *
import MySQLdb as mdb
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
import random



boardNum=0
portNum=10
inputVoltage = 3



BoardNum = 0
Gain = BIP5VOLTS
Chan = 0



spTimes =14*24*60*60     ##here you set the time
timeDuration=spTimes
padsToVisualize =[0,1,2,3,4]  ##pads that will be measured and visualized

#plt.ion()    ##this command is necassary in order to have new points on the plot without erasing the old ones

curTime=time.clock()
movingWinDur=60    ##seconds
print "curtime=",curTime

str_one_line=[]
straverage=[]
while (curTime < timeDuration):
    del str_one_line[:]
    padCounter=0;

    d=datetime.datetime.now()
    str_one_line=[(d.strftime("%Y-%m-%d %H:%M:%S")),',']   ##formatting the date -- this format is chosen based on the requirements for connecting to SQL database

    while (padCounter <len(padsToVisualize)):

        #fig1=plt.figure(padsToVisualize[padCounter])    ## making figures for plotting different pads


        print "pad No.: ", str(padsToVisualize[padCounter])
        #l=cbAOut(BoardNum,Chan,Gain,3019)   ## sets the voltage to 3.0 volts (the number 3019 is based on 12-bit resolution --> DAQ resolution)
        #n=cbDConfigPort(BoardNum,portNum,1)   ##the variable n serves aas a command line where the command is sent to the DAQ
        #b=cbDOut(BoardNum,portNum,(63-padsToVisualize[padCounter])) #####based on Prof. Mayer's solution, the line sends the pad number that we are interested in to the DAQ to get the signals 
        curTime=time.clock()
        d=datetime.datetime.now()
        #########################################################################

        ########## this part will use the datetime to fetch the temperature from the SQL data base
        ########## since the time stamps may not exactly match, the code uses a method of interpolation
        freq = 5
        time.sleep(1.8*freq) ## 1.8 is an arbitrary number ; to make sure there is data on sql database
        searchfor=d   ## the following lines find the temperature at time=searchfor
        print searchfor
        minussearchfor = searchfor-datetime.timedelta(0,freq/1.2)
        STRminussearchfor = minussearchfor.strftime("%Y-%m-%d %H:%M:%S")
        print "STRminussearchfor=", STRminussearchfor
        print "minussearchfor= ", minussearchfor
        plussearchfor =searchfor+datetime.timedelta(0,freq/1.2)
        print "plussearchfor= ", plussearchfor
        STRplussearchfor = plussearchfor.strftime("%Y-%m-%d %H:%M:%S")
        print "STRplussearchfor=", STRplussearchfor
        ##Database connection
        db = mdb.connect("localhost", "root", "qwerty@12345", "bondtest")
        cur = db.cursor()
        cur.execute("SELECT * FROM bondtest_data WHERE DateTime BETWEEN %(date1)s AND %(date2)s ORDER BY DateTime",{'date1':STRminussearchfor,'date2':STRplussearchfor})
##        con=pymssql.connect(host='LAB10-PC\SQLEXPRESS2008R2',user='sa',password='qwerty@12345')
##        cur = con.cursor()
##        cur.execute('SELECT * FROM OVEN11SQL.dbo.OvenLog1 WHERE DateTime BETWEEN %(date1)s AND %(date2)s ORDER BY DateTime',{'date1':STRminussearchfor,'date2':STRplussearchfor})
        tupline1 = cur.fetchone()
##        print 'between1= ',tupline1[1]
        delta = tupline1[1]-searchfor
##        print "delta = " ,delta
        if (int(searchfor.strftime("%S"))-int(tupline1[1].strftime("%S"))>0):
            delta = int(searchfor.strftime("%S"))-int(tupline1[1].strftime("%S"))
        else:
            delta = int(tupline1[1].strftime("%S"))-int(searchfor.strftime("%S"))
##        print 'delta= ',delta
        time1=tupline1[1]-datetime.timedelta(0,freq/1.2)
        STRtime1=time1.strftime("%Y-%m-%d %H:%M:%S")
        time2=tupline1[1]-datetime.timedelta(0,3*freq/1.2)
        STRtime2=time2.strftime("%Y-%m-%d %H:%M:%S")
##        time.sleep(2*freq) ##the program needs to wait for 3*frequency/2 so that the next point is available in SQL data base for interpolation
        cur.execute('SELECT * FROM bondtest_data WHERE DateTime BETWEEN %(date1)s AND %(date2)s ORDER BY DateTime',{'date1':STRtime2,'date2':STRtime1})
        tupline2 = cur.fetchone()   ##next point is fetched in order to find the slope of the line for temperature change
##        print 'between2= ', tupline2[1]
        deltay=float(float(tupline1[3])/10-float(tupline2[3])/10)
        deltax = int(tupline1[1].strftime("%S"))-int(tupline2[1].strftime("%S"))
        deltax = freq
##        print "deltay= ", deltay
##        print "deltax= ", deltax
        slope = deltay/deltax
##        print 'slope= ', slope

        ##in the following if statements depending on whether the temperature is increasing or decreasing the temperature for the desired point will be calculated
        if (tupline2[3]<tupline1[3]):
            tempsearchfor = float(tupline1[3])/10+delta*slope
##            print '+delta*slope= ',delta*slope
        elif (tupline2[3]>tupline1[3]):
            tempsearchfor = float(tupline1[3])/10-delta*slope
##            print '-delta*slope= ',delta*slope
        else:
            tempsearchfor = float(tupline1[3])/10
##        print 'tempserachfor= ',tempsearchfor

        #########################################################################

        strng = [str(int(padsToVisualize[padCounter])),',',(d.strftime("%Y-%m-%d %H:%M:%S")),',',str(round(curTime,4)),',',str(inputVoltage),',']
        str_one_line.extend(strng)         ##str_one_line is the line that contains the values that will be written to the text file , the order is specified in the variables lists excel file
        xyzCounter=Chan
        EngUnits= array(range(50*1),dtype=float).reshape(50,1)    ## constructing the array that will hold the 50 values for each set of signals and is later used for finding the average
        average = array(range(3*1),dtype=float).reshape(3,1)      ## holds the average of the t50 points for x,y,z
##        straverage=array(range(3*1),dtype=str).reshape(3,1)   ##this array will hold the strings for x,y,z signal to be written into txtfile
        del straverage[:]
        while(xyzCounter<3):   ##the way the oven is set up it has x,y,z outputs from channel 0,1,2 accordingly
            #n=cbDConfigPort(BoardNum,portNum,1)
            #b=cbDOut(BoardNum,portNum,(63-padsToVisualize[padCounter]))     #####based on Prof. Mayer's solution, the line sends the pad number that we are interested in to the DAQ to get the signals
            a=0
            time1=time.clock()
            while (a<50):    ## this while loop takes 50 measurements and writes the average of those points in to average array

                #DataValue=cbAIn(BoardNum, xyzCounter, Gain)
                #EngUnits[a,0]=float((cbToEngUnits(BoardNum,Gain,DataValue))/3/100*1000)
                EngUnits[a,0] = random.uniform(0,0.5)
                average[xyzCounter,0]=float(sum(EngUnits))/len(EngUnits)
                a+=1
            time2=time.clock()
            timePerSample=(time2-time1)/50
            print "time per sample= ",timePerSample
            print "samples per second =", (1/timePerSample)   ##measuring the frequency of the emasurements
            tempstr=str(round((average[xyzCounter,0]),3))    ##in order to remove the two brackets temp refers to temporary
            #tempstr=tempstr[1:-1]
            straverage.append(tempstr)
            xyzCounter+=1


        #print average
        temperaturearray=array(range(1*1),dtype=str).reshape(1,1)
        temperaturearray=[str(tempsearchfor)]
        three_sens_signals=array(range(1*5),dtype=str).reshape(1,5)
        three_sens_signals=[str((straverage[0])),',',str((straverage[1])),',',str((straverage[2])),',']
        str_one_line.extend(three_sens_signals)
        str_one_line.extend(temperaturearray)
        str_one_line.extend(',')

        padCounter+=1
        filename='log_simulation.txt'
        f = open(filename,"a")
      ## writing to file section
    print "padcounter=",padCounter,"   str_one_line=", str_one_line
    for item in str_one_line:  
        f.write("%s" % item )
    f.write("\n")
    f.close()

    curTime=time.clock()

Answers


In short, there are a lot of things that need to be improved/reworked in this code (start by editing all the import * statements and using namespace like you should). Having said that (the code apparently mostly works as is), the issue occurs because the value stored in tupline or tupline2 is None for whatever reason. Further up in your code these variables are assigned using the sql statements

tupline = cur.fetchone()
tupline2 = cur.fetchone() 

One of those calls is apparently not running smoothly (for whatever reason the data or table is corrupt) which causes it to return "None". Maybe someone is deleting or replacing the table? You can add some semantics to check for this case, report and try to continue. Something along the lines of

if tupline == None or tupline2 == None:
    print "Why does tupline = None?"
    continue

You should figure out when sql .fetchone() method returns none. Really you should do some better logging but that might get you through to the next step of debugging.


As Paul Seeb noted, the error message occurs because tupline or tupline2 is None when the code reaches the deltay=... line. Your code (successfully) accesses elements of tupline after it is assigned, before reaching the deltay line. So the problem must be with tupline2.

My guess would be that the database contains no matching records for the tupline2 query; that would certainly be something to check out. I also note that you have commented out a delay just before the tupline2 query which hints that this query doesn't reliably return records. I don't know your program logic, but you might be able to fix the problem with lines like this just after the tupline2 query:

if tupline2 is None:
    continue`

Or you might retry the tupline2 query until it succeeds

while True:
    cur.execute(
        """SELECT * FROM bondtest_data 
            WHERE DateTime BETWEEN %(date1)s AND %(date2)s 
            ORDER BY DateTime""",
        {'date1':STRtime2,'date2':STRtime1}
    )
    tupline2 = cur.fetchone()
    if tupline2 is None:
        print "Failed to obtain a record for tupline2; trying again..."
        time.sleep(<some duration>)
    else:
        break

I also note that your code creates a new database connection (db) every time it goes through the loop and never explicitly closes it. I would recommend moving the db = mdb.connect(...) command above the start of the main loop, or adding db.close() somewhere after the last fetchone() command. It is distantly possible that your code is reaching some kind of connection limit, but that would probably raise a different exception. More likely you are benefiting from some kind of automatic closing of the connection when db gets reassigned and the old connection goes out of scope.


Need Your Help

Remain within magnific popup

ajax lightbox magnific-popup

I have an ajax popup working nicely with Magnific Popup. However the page that I am loading into the popup via the ajax call contains a hyperlink.

Drupal - Views Slideshow

javascript jquery drupal drupal-7

I am working with views slideshow on Drupal to display a image per page on a designated block. The issue I am experiencing is that the module is cycling through different images. I want to somehow ...