Sunday 22 February 2015

Ultra Sonic Pinky Pig Anti-Theft Device

Meet Pinky Pig:



Pinky Pig is the most loved soft toy in the Geek household.  He's been owned by my eldest daughter since birth, is a a bit worse for wear these days but is loved all the same.  In fact Pinky Pig rarely leaves the house,such would be the trauma of losing him.

In a search for something new to do with our ultra sonic sensor (see last week's post), we came up with the idea of using the ultra sonic sensor to make a device to detect if Pinky has been stolen.  In simple terms, Pinky's resting place would be on the ultra sonic sensor.  If he was taken then this could be detected and an alarm raised.  

We had a lot of fun (and a lot of learning) coming up with the Scratch code below on one of our Raspberry Pi's.  It was a really good way to focus on the role of a loop and an if statement.



So if the measurement from the sensor is small, all is well and no alarm is raised.  However if the measurement is not small then the (audible) alarm is raised.

We we did learn which was interesting was that Pink Pig (and other cuddly toys) are transparent to ultra sonic sound waves.  Very interesting, this added an extra layer of investigation to this project.  Please comment below if you can tell me why this is the case.  Hence to make the alarm work, Pinky had to be wrapped in an extra layer of paper.  Here's a picture showing this in action.


So last thing, here's a video of it in action (wish the volume on the TV was louder - listen carefully and you can hear the alarm going off).



Monday 16 February 2015

Raspberry Pi Ultra Sonic Press Up (Push Up) Counting Machine using Scratch

Looking at back at recent posts I've noticed:
  • I've not done any electronics related projects recently.
  • I've not done much programming with my kids recently.
When doing some press ups* recently I lost count and couldn't log them on my Jerks exercise logging system.  Hence this gave me the idea to build a machine to count my press ups.

(*Push ups for you readers in North America)

I saw you can buy one from Amazon and there are some available as smart phone apps but this isn't the Geek Dad way; I wanted to build one with a Raspberry Pi, program it in Scratch and use some electronics.  Ideas:
  • Mount a light dependent resistor - as I go down it blocks out light from the LDR, as I go up it allows light again and counts a press up.
  • Use a camera and a QR code.  As I go down it reads the QR code and counts a press up.
  • Use an ultra sonic sensor to measure distance as I go up and down.
I decided to use an ultra sonic sensor - purely because I'd hadn't used one since a University project about 20 years ago.  Here's the one I bought, an HC-SR04 module which has both a transmitter and a receiver.  They're widely available on the interweb.


To integrate the sensor to my Raspberry Pi I used the utterly amazing, epic and awesome Cymplecy Scratch GPIO system (did I mention that this site is brilliant).  This chap has developed a series of Scratch releases that allow you to integrate with many and various third party boards and bits of electronic kit that you can connect to the GPIO.  There's a specific page on Ultra Sonic module integration; it's available here and the solution only uses one GPIO input/output which is ace.

The resistors shown on the breadboard diagram on Cymplecy's site are essential to stop you blowing up your Raspberry Pi; they create a potential divider to reduce the +5V output from the HC-SR04 to a GPIO friendly 3.3V.  The resistors are:
  • Brown-Black-Red = 1kOhms
  • Red-Red-Red = 2.2kOhms
I won`t do the maths here, but see this site for an explanation of potential dividers.

The system works brilliantly and with this code block you can get Scratch to show the measurement from the ultra sonic sensor and move the Scratch Cat around.



So this worked beautifully, next I needed a robust housing for the press up counting machine to avoid me breaking it.  Enter Lego, the tool of tinkerers everywhere.  Here's a series of before, during and after shots:





So the breadboard is mounted perpendicular in the Lego housing, meaning the sensor points straight up.  Hence as I'm doing press ups I'm moving up and down vertically over the sensor.  The Raspberry Pi sits separate to it.

Here's a quick video:



The basic algorithm for the press up code is as follows:

boolean = false
forever
  Take Measurement
  if Measurement < Low AND boolean = false
    boolean = true
  if Measurement > High and boolean = true
    increment press up count
    boolean = false

So this uses a simple boolean to log when my body is low down (say < 10cm from the sensor).  The spots the bottom part of the press up.  Then when I move up, if my body is high up (say > 30cm) AND I've previously been low down then a press up is counted (and the boolean is reset).

The basic Scratch code is shown on the screen shot below (put your email address in a comment below and I'll send you the file).  This does a good job of counting press ups (although you have to go quite slow as the sampling period is only one second) and getting the Scratch Cat to say them.



What I wanted to do with my daughters is jazz it up a bit.  In preparation for this I recorded a number of different audio messages for Scratch to play out.  I used this site which takes a text string input and provides a downloadable MP3 file.  These files can then be put in the /usr/share/scratch/Media/Sounds directory on your Raspberry Pi.  Armed with this I spent an enjoyable afternoon with my daughters making the press up counter more interesting.

We:
  • Changed the background to make it look better.
  • Added a sound at the start to tell you that "It's press up time!".
  • Added If statements to detect each Press Up (up to 10) and play an audio message with the number ("1" for the first press up, "5" for the fifth etc).
  • Added some motivational (semi-abusive) messages to keep you going.
  • Added a "Well Done" message at the end.

Here's a video of it in action!



Here's a screenshot of the Scratch code (top part). A but hard to see so, again, put your email address in comments below and I'll send you the Scratch file.


...and bottom part:







    

Saturday 7 February 2015

Fixing LightwaveRF UDP Power Measurements

The thing that got me into this Geek blogging lark was a challenge from an old boss to look into how home automation equipment works.  This culminated in a posting on how to take power measurements on a Raspberry Pi and post them to an online measurements logging site (COSM, now called Xively).

Those measurements were happily up, working and logging for over 3 years until they abruptly stopped working towards the end of last year.  Looking at the LightwaveRF control unit (the so called WiFi link) the firmware version is now 2.91Q.  A bit of interweb searching tells me that this version was introduced at some point in late 2014 and I'm guessing it's that which has caused the problem.  (In an old posting I noticed that I had version 2.28).

It's taken me a while to get round to it  but I've finally investigated and fixed the problem that stopped the measurements working.  Here's what I did...

Step 1 was to check the IP address used for the LightwaveRF control unit.  Reminder: the way to get measurements is to send the string "123,@?\0" in a UDP segment to port 9760 on  the IP address that the LightwaveRF control unit sits on.  The response comes back on port 9761.

The IP address of the LightwaveRF control unit is defined as a constant in my script so I had to make sure that the one I had defined was still the correct one.  A quick check of my router "LAN IP Address" settings (which means the same IP address is always allocated to the same MAC address) showed that it was all OK.

Step 2 was to use tcpdump to work out what was going on on my LAN between my Raspberry Pi and the LightwaveRF control unit.

The command sudo tcpdump -n -i any -s0 -vv -X host 192.168.0.2 means:

"-n" means don't resolve addresses, "-i any" means any interface, "-s0" means all the packet, "-vv" means very verbose, "-X" means give us the full packet.

The log I got was:

15:31:24.446184 IP (tos 0x0, ttl 64, id 3918, offset 0, flags [DF], proto UDP (17), length 35)
    192.168.0.5.46246 > 192.168.0.2.9760: [udp sum ok] UDP, length 7
        0x0000:  4500 0023 0f4e 4000 4011 aa24 c0a8 0005  E..#.N@.@..$....
        0x0010:  c0a8 0002 b4a6 2620 000f ff13 3132 332c  ......&.....123,
        0x0020:  403f 00                                  @?.
15:31:24.651793 IP (tos 0x0, ttl 100, id 63682, offset 0, flags [none], proto UDP (17), length 56)
    192.168.0.2.9760 > 255.255.255.255.9761: [udp sum ok] UDP, length 28
        0x0000:  4500 0038 f8c2 0000 6411 9d48 c0a8 0002  E..8....d..H....
        0x0010:  ffff ffff 2620 2621 0024 38ed 3132 332c  ....&.&!.$8.123,
        0x0020:  3f57 3d35 3739 2c39 3536 312c 3632 3531  ?W=579,9561,6251
        0x0030:  2c35 3630 343b 0d0a                      ,5604;..
15:31:27.110984 IP (tos 0x0, ttl 100, id 63683, offset 0, flags [none], proto UDP (17), length 196)
    192.168.0.2.4096 > 255.255.255.255.9761: [udp sum ok] UDP, length 168
        0x0000:  4500 00c4 f8c3 0000 6411 9cbb c0a8 0002  E.......d.......
        0x0010:  ffff ffff 1000 2621 00b0 a208 2a21 7b22  ......&!....*!{"
        0x0020:  7472 616e 7322 3a32 3631 3335 362c 226d  trans":261356,"m
        0x0030:  6163 223a 2230 333a 3031 3a35 4222 2c22  ac":"03:01:5B","
        0x0040:  7469 6d65 223a 3134 3232 3830 3436 3836  time":1422804686
        0x0050:  2c22 7072 6f64 223a 2270 7772 4d74 7222  ,"prod":"pwrMtr"
        0x0060:  2c22 7365 7269 616c 223a 2242 4332 4246  ,"serial":"BC2BF
        0x0070:  4522 2c22 7369 676e 616c 223a 3535 2c22  E","signal":55,"
        0x0080:  7479 7065 223a 2265 6e65 7267 7922 2c22  type":"energy","
        0x0090:  6355 7365 223a 3537 372c 226d 6178 5573  cUse":577,"maxUs
        0x00a0:  6522 3a39 3536 312c 2274 6f64 5573 6522  e":9561,"todUse"
        0x00b0:  3a36 3235 332c 2279 6573 5573 6522 3a35  :6253,"yesUse":5
        0x00c0:  3630 347d                                604}

The first packet is the command sent from my Python script.

The second and third packets are from the LightwaveRF control unit.  There are two points to notice:

1)The response is now sent to 255.255.255.255, the zero network broadcast address.  I assume that this means you can have several hosts on the same network logging and broadcasting their results.

2)There is now two responses from the control unit.  One looks to be in the same format as before (four comma separated values) and the other looks to be some longer format that needs investigating.

However what I also spotted was that the responses in the new format seem to come without me actually sending the "123,@?\0" string.  This was puzzling until I realised that the control unit was broadcasting out measurements without being prompted, i.e. without me sending anything in the first place.  A simpler tcpdump shows:

pi@raspberrypi ~/lightwaverf $ sudo tcpdump -n -i any -s0 host 192.168.0.2
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
18:26:36.135825 IP 192.168.0.2.4096 > 255.255.255.255.9761: UDP, length 169
18:26:51.291677 IP 192.168.0.2.4096 > 255.255.255.255.9761: UDP, length 169
18:27:06.344437 IP 192.168.0.2.4096 > 255.255.255.255.9761: UDP, length 169
18:27:21.501676 IP 192.168.0.2.4096 > 255.255.255.255.9761: UDP, length 169
18:27:36.553111 IP 192.168.0.2.4096 > 255.255.255.255.9761: UDP, length 169
18:27:51.708639 IP 192.168.0.2.4096 > 255.255.255.255.9761: UDP, length 169
18:28:06.761778 IP 192.168.0.2.4096 > 255.255.255.255.9761: UDP, length 169
18:28:21.917158 IP 192.168.0.2.4096 > 255.255.255.255.9761: UDP, length 169

Here you can see a measurement being broadcast every 15 seconds.

So these are the problems that need fixing....

Problem 1 was easy to fix.  Changing a constant to:

UDP_IP = "255.255.255.255"

Means that when these commands are run:

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # UDP
sock.bind((UDP_IP, UDP_PORT))

...the script binds to 255.255.255.255 to capture the response.

Problem 2 was a bit harder to fix.  Capturing and printing the new string showed it to be of this format:

*!"trans":261534,"mac":"03:01:5B","time":1422813309,"prod":"pwrMtr","serial":"BC2BFE","signal":55,"type":"energy","cUse":1504,"maxUse":9561,"todUse":7696,"yesUse":5604}

So a series of comma and colon separated values.  The initial values look to be useful metadata (e.g. MAC address, power meter etc) which would be useful in a multi-measurement, multi-unit environment.  However for me it's just the last 4 elements I need (current power, max power used today, cumulative today and cumulative yesterday).

I updated the code to remove the old measurement handling statements and to interpret the new measurement types.

The code below is the key new element added.  I also removed the code handling the old response format.  As the control unit just broadcasts energy measurements I no longer have to run a script to poll for information.  This should make the overall system more reliable...

#Put result into an array and get rid of the superfluous bits
      ResultArray = data.split(',')
      MeasurementCSV = TrimAndTidy(ResultArray[7]) + ',' + TrimAndTidy(ResultArray[8]) + ',' + TrimAndTidy(ResultArray[9]) + ',' + TrimAndTidy(ResultArray[10])
      print 'Measurement CSV >>> ' + MeasurementCSV

Here I simple split the string into parts (based upon the comma) and then send it to a tidy up function.  This is a simple piece of code:

def TrimAndTidy(InputString):
  #First split based upon the : and then remove a }
  try:
    ArrayOfVals = InputString.split(':')
    return ArrayOfVals[1].replace('}','')
  except:  #Just return a zero
    return '0'

That simply splits based upon the colon and cleans up the extra } at the end of the response string.

As an aside, one of the thing the code does is log to a local log file (in CSV format).  The log file is 32Meg in size and will have several hundred thousand rows of data it.  I smell some excellent geek action analysing those files to look for trends....

Here it is working again, huzzah!



...and here's all the code:

#v1=Just Watts measurement.  V2=Added other4 values.  V3=Added  GBP values.  V4=Added errorlogging.  V5=Handled lightwaverf response format changes
#V6=Removed old methodology

#Change IP address un comment chmod before putting on the phone "server"

#Import statements
import socket
import datetime
from os import chdir
import httplib
import sys

#Some constants for this, the server (192.168.0.5)
UDP_IP = "255.255.255.255" #Changed 31/01/2015 when Lightwaverf box started sending responses to broadcast address
UDP_PORT = 9761            #Responses always sent to this port

#These are constants related to the COSM feed
TimeZoneOffset = "+00:00"
MyFeedID = "YOUR_ID_HERE"
MyWattsNowAPIKey = "YOUR_KEY_HERE"
MyWattsCostAPIKey ="YOUR_KEY_HERE"
MyWattsNowDataStream = "WattsNow"
MyMaxWattsDataStream = "WattsNow_Max"
MyCumulativeWattsDataStream = "WattsNow_Cumulative"
MyYesterdayTotalDataStream = "WattsNow_TotalYesterday"
MyCumulativeCostDataStream = "WattsNow_CostToday"
MyCostYesterdayDataStream = "WattsNow_CostYesterday"


#Constants related to costs
UnitCost =11.68
DailyCost = 18.00

#This is a Python function that writes a log file.  Used for debugging purposes
def WriteDebugLog(StrToLog):
  #Form a date and time for this
  #Get the date and time
  DateToday = datetime.date.today()
  TimeNow = datetime.datetime.now()

  #Form the string we will write to screen and local file
  LogFileString = str(DateToday) + "," + str(TimeNow) + "," + StrToLog


  #And log to file.  "a" means append if necessary
  #COMMENT OUT THE 3 ROWS BELOW ELSE UNLESS YOU NEED THE LOG
  logfile = open("energy_measurements_log_file.txt", "a")
  logfile.write(LogFileString + "\n")
  logfile.close()
  return

#This is a Python function to log to COSM
def SendToCOSM(ValToSend,KeyToUse,FeedToUse,DataStreamToUse):
  #Use this try statement to capture errors
  try:
    #Write to our debug log file
    WriteDebugLog("Start of write to COSM Function. " + DataStreamToUse)

    #First form the string to send.  Here be an example '2012-09-30T22:00:00.676045+01:00,66'
    #So we need some date geekery for this
    #Get a variable to hold the date
    today = datetime.datetime.now()

    #Create an overall string with the story so far
    MyDateTimeString = today.strftime("%Y-%m-%d") + "T"

    #Now for the time bit - First the format string
    FormattedTime = today.strftime("%H:%M:%S")    #Get the formatted time

    #Now form the full monty string
    MyDateTimeString = MyDateTimeString + FormattedTime + TimeZoneOffset + "," + ValToSend

    #And get it's length
    MyStrLen = str(len(MyDateTimeString))

    #Print what we got so far
    print 'FullString:', MyDateTimeString

    #Now do the HTTP magic - Connect to the server
    h = httplib.HTTP('api.cosm.com')

    # build url we want to request
    FullURL = 'http://api.cosm.com/v2/feeds/'+ FeedToUse + '/datastreams/' + DataStreamToUse + '/datapoints.csv'

    #Print the URI string we will use
    print "Full URL: " + FullURL

    # POST our data.
    h.putrequest('POST',FullURL)

    # setup the user agent
    h.putheader('X-ApiKey',KeyToUse)
    h.putheader('Content-Length',MyStrLen)

    # we're done with the headers....
    h.endheaders()

    #Send the data
    h.send(MyDateTimeString)

    #Get the response from the request
    returncode, returnmsg,headers = h.getreply()

    #display whatever the results are....
    f = h.getfile()
    MyData = f.read()
    print f.read()

    #Write to our debug log file
    WriteDebugLog("End of write to COSM Function")

    #Now just return
    return
  #Catch an exception
  except Exception, err:
    #Write a log with the error
    print "Got us an exception: " + str(err)
    #WriteDebugLog("Caught this error in log to COSM function: " + str(err)

#This function calculates the cost in pounds for the electricity used.
#The formula is ((WattHours/ 1000) * (UnitCost / 100)) + (DailyCharge / 100)
def CalculateCosts(InWattHours):
  #WattHours comes in as a string so need to turn to a number

  #Added 26/10/2014 - First check InWattHours
  print "InWattHours: " + InWattHours

  if (";" in InWattHours):
    print "Caught a rogue InWattHours: " + InWattHours
    CostInPoundsFloat = "0"
  else:
    #do the calculation
    CostInPoundsFloat = ((float(InWattHours) / 1000) * (UnitCost / 100)) + (DailyCost / 100)

    #Round it to 2 decimal places
    CostInPoundsFloat = round(CostInPoundsFloat,2)

  #return a string
  return str(CostInPoundsFloat)

#New def added 01/02/2015 to handle the new format of command response.  Example
#*!{"trans":261395,"mac":"03:01:5B","time":1422806123,"prod":"pwrMtr","serial":"BC2BFE","signal":57,"type":"energy","cUse":809,"maxUse":9561,"todUse":6349,"yesUse":5604}
#Will get one of the pieces from within the commas.  Need to get the part after the colon and if required trim off a }
def TrimAndTidy(InputString):
  #First split based upon the : and then remove a }
  try:
    ArrayOfVals = InputString.split(':')
    return ArrayOfVals[1].replace('}','')
  except:  #Just return a zero
    return '0'

########################################
#Now we start the main part of the code
########################################


#Change directory that we will write to
chdir('/home/pi/lightwaverf')

#Tell the user we've started
print "UDP server started.  Waiting for response...."

#Bind a socket
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # UDP
sock.bind((UDP_IP, UDP_PORT))

#Now just loop unit you receive a response
while True:
    #Read data from the buffer
    data, addr = sock.recvfrom(1024) #buffer size is 1024

    #Write to our debug log file
    WriteDebugLog("What we read from the buffer: " + data)

    #A debug print
    print 'I just read this >>> ' + data

    try:
      #Put result into an array and get rid of the superfluous bits
      ResultArray = data.split(',')
      MeasurementCSV = TrimAndTidy(ResultArray[7]) + ',' + TrimAndTidy(ResultArray[8]) + ',' + TrimAndTidy(ResultArray[9]) + ',' + TrimAndTidy(ResultArray[10])
      print 'Measurement CSV >>> ' + MeasurementCSV

      #Write to our debug log file
      WriteDebugLog("Just the measurements after removing the command prefix: " + MeasurementCSV)

      #Get the date and time
      today = datetime.date.today()
      TheTime = datetime.datetime.now()

      #Form the string we will write to screen and local file
      OutString = str(today) + "," + str(TheTime) + "," + MeasurementCSV

      #Print the result...
      print OutString

      #Write to our debug log file
      WriteDebugLog("The string that we will log to the log file: " + OutString)

      #And log to file.  "a" means append if necessary
      logfile = open("energy_measurements.csv", "a")
      logfile.write(OutString)
      logfile.close()

      #Write to our debug log file
      WriteDebugLog("Have just written the log file CSV")

      #Split the string and assign to variables
      SplitMeasurement = MeasurementCSV.split(',')
      WattsNow = SplitMeasurement[0]            #The power value for now (Watts)
      MaxWatts = SplitMeasurement[1]            #The max power today (Watts)
      CumToday = SplitMeasurement[2]            #Cumulative today (Watt Hours)
      TotalYesterday = SplitMeasurement[3]      #Total yesterday (Watt Hours)

      #Write to our debug log file
      WriteDebugLog("Have just split the string in 4")

      #Print the output
      print "Watts Now [W]:" + WattsNow
      print "Max Watts Today [W]:" + MaxWatts
      print "Cumulative Today [Wh]:" + CumToday
      print "Total Yesterday [Wh]:" + TotalYesterday

      #Write to our debug log file
      WriteDebugLog("Have just printed the measurements to screen")

      #Log to COSM dude!!! First check it's not 0 as that looks rubbish!
      if WattsNow == "0":
        print "Not sending as it's 0 Watts"

        #Write to our debug log file
        WriteDebugLog("Saw that the Watts measurement was 0 so didn't log to COSM")
      elif (WattsNow.isdigit() == False):
        print "Not sending as it's not a numeric Watts value"

        #Write to our debug log file
        WriteDebugLog("Saw that the Watts measurement was not a number so didn't log to COSM.  Measurement >> " + WattsNow)
      else:
        SendToCOSM(WattsNow,MyWattsNowAPIKey,MyFeedID,MyWattsNowDataStream)
        SendToCOSM(MaxWatts,MyWattsNowAPIKey,MyFeedID,MyMaxWattsDataStream)
        SendToCOSM(CumToday,MyWattsNowAPIKey,MyFeedID,MyCumulativeWattsDataStream)
        SendToCOSM(TotalYesterday,MyWattsNowAPIKey,MyFeedID,MyYesterdayTotalDataStream)

        #Write to our debug log file
        WriteDebugLog("Have just sent the 4 measurements to COSM.  Now calculate costs.")

        #Now calculate the costs
        CumulativeCost = CalculateCosts(CumToday)
        TotalYesterdayCost = CalculateCosts(TotalYesterday)

        print "Cumulative Cost GBP" + CumulativeCost
        print "TotalCost GBP" + TotalYesterdayCost

        #Write to our debug log file
        WriteDebugLog("Have calculated costs. Was cumulative GBP" + CumulativeCost + "and yesterday GBP" + TotalYesterdayCost + ". Now send to COSM")

        #Send them to COSM
        SendToCOSM(CumulativeCost,MyWattsCostAPIKey,MyFeedID,MyCumulativeCostDataStream)
        SendToCOSM(TotalYesterdayCost,MyWattsCostAPIKey,MyFeedID,MyCostYesterdayDataStream)

        #Write to our debug log file
        WriteDebugLog("Sent costs to COSM.")
    except Exception, err:
      #Write a log with the error
      print "Got us an exception: " + str(err)
      #WriteDebugLog("Caught this error in the main area of code: " + str(err)