Further Development of Video Based Seizure Detector

I have made a bit more progress with the video based epileptic seizure detector.

Someone on the OpenCV Google Plus page suggested that I look at the Lucas-Kanade feature tracking algorithm, rather than trying to analyse all of the pixels at once like I was doing.

This looks quite promising.  First you have to decide which features in the image to use - corners are good for tracking.  OpenCV has a neat cv.GoodFeaturesToTrack function which makes suggestions - you give it a couple of parameters, including a 'quality' parameter to help it choose.  This gives a list of (x,y) coordinates of the good features to track.  Note that this means 'good' mathematically, not necessarily the limbs of the test subject....

Once you have some features to track, OpenCV again provides a cv.CalcOpticalFlowPyrLK, where you give it the list of features, the previous image and a new image, and it calculates the locations of the features in the new image.

I have then gone into the fourier analysis that I have been trying for the other types of seizure detection. This time I calculate the speed of each feature over a couple of seconds, and record this as a time series, then calculate the fourier transform to give the frequency spectrum of the motion.   If there is oscillation above a threshold amplitude in a given frequency band for a specified time we raise an alarm as a possible seizure.

The code is functioning, but is a fair way off being operational yet.  The code for this is in my OpenSeizureDetector github repository (https://github.com/jones139/OpenSeizureDetector).

The current issues are:

• I really want to track motion of limbs, but there is no guarantee that cv.GoodFeaturesToTrack will detect these as good features - I can make this more likely by attaching reflective tape, which glows under IR illumination from the night vision camera...if I can persuade Benjamin to wear it.
• There is something wrong with the frequency calculation still - I can understand a factor of two, but it seems a bit more than that.
• If the motion is too quick, it looses the point, so I have to set it to re-initialise using GoodFeaturesToTrack periodically.
• An Example of it working with my daughter doing Benjamin-like behaviour is shown below.   Red circles are drawn around points if a possible seizure is detected.

• This does not look too good - lots of points detected, and even the reflective strips on the wrists and ankles get lost.  It seems to work better in darkness though, where I get something like the second video, where there are only a few points, and most of those are on my high-vis reflective strips.

• It does give some nice debugging graphs of the speed measurements and the frequency spectra though.

So, still a bit of work to do.....

First go at a Video Based Epileptic Seizure Detector

Background

I have been working on a system to detect epileptic seizures (fits) to raise an alarm without requiring sensors to be attached to the subject.
I am going down three routes to try to do this:

• Accelerometers
• Audio
• Video

This is about my first 'proof of concept' go at a video based system.

Approach

I am trying to detect the shaking of a fit.  I will do this by monitoring the signal from an infrared video camera, so it will work in monochrome.  The approach is:

1. Reduce the size of the image by averaging pixels into 'meta pixels' - I do this using the openCV pyrDown function that does the averaging (it is used to build image pyramids of various resolution versions of an image).  I am reducing the 640x480 video stream down to 10x7 pixels to reduce the amount of data I have to handle.
2. Collect a series of images to produce a time series of images.  I am using 100 images at 30 fps, which is about 3 seconds of video.
3. For each pixel in the images, calculate the fourier transform of the series of measured pixel intensities - this gives the frequency at which the pixel intensity is varying.
4. If the amplitude of oscillation at a given frequency is above a threshold value, treat this as a motion at that particular frequency (ie, it could be a fit).
5. The final version will check that this motion continues for several seconds before raising an alarm.  In this test version, I am just  highlighting the detected frequency of oscillation on the original video stream.

Code

The code uses the OpenCV library, which provides a lot of video and image handling functions - far more than I understand...

My intention had been to write it in C, but I struggled with memory leaks (I must have been doing something wrong and not releasing storage, because it just ate all my computer's memory until it crashed...).

Instead I used the Python bindings for OpenCV - this ran faster and used much less memory than my C version (this is a sign that I made mistakes in the C one, rather than Python being better!).

The code for the seizure detector is here - very rough 'proof of concept' one at the moment - it will have a major rewrite if it works.

Test Set Up

To test the system, I have created a simple 'test card' video, which has a number of circles oscillating at different frequencies - the test is to see if I can pick out the various frequencies of oscillation.  The code to produce the test video is here....And here is the test video (not very exciting to watch I'm afraid).

The circles are oscillating at between 0 and 8 Hz (when played at 30 fps).

Results

The output of the system is shown in the video below.  The coloured circles indicate areas where motion has been detected.  The thickness of the line and the colour shows the frequency of the detected motion.

• Blue = <3 hz="" li="">
• Yellow = 3-6 Hz
• Red = 6-9 Hz
• White = >9 Hz

The things to note are:

• No motion detected near the stationary 0 Hz circle (good!).
• <3hz 1="" 2="" and="" circles="" detected="" good="" hz="" li="" motion="" near="" the="">
• 3-6 Hz motion detected near the 2,3,4 and 5 Hz circles (ok, but why is it near the 2Hz one?)
• 6-9 Hz motion detected near the 5 and 6 Hz circles (a bit surprising)
• >9Hz motion detected near the 4 and 7 Hz circles and sometimes the 8Hz one (?)

So, I think it is sometimes getting the frequency too high.  This may be as simple as how I am doing the  check - it is using the highest frequency that exceeds the threshold.  I think I should update it to use the frequency with maximum amplitude (which exceeds the thershold).

Also, I have something wrong with positioning the markers to show the motion - I am having to convert from a pixel in the low res image to the location in the high resolution one, and it does not always match up with the position of the moving circles.

But, it is looking quite promising.  Rather computer intensive at the moment though - it is using pretty much 100% of one of the CPU cores on my Intel Core I5 laptop, so not much chance of getting this to run on a Raspberry Pi, which was my intention.

Getting Started with OpenCV

I am starting work on the video version of my Epileptic Seizure detector project, while I wait for a very sensitive microphone to arrive off the slow boat from China, which I will use for the Audio version.

I am using the OpenCV computer vision library.  What I am hoping to do is to either:

• Detect the high frequency movement associated with a seizure, or
• Detect breathing (and raise an alarm if it stops)

This seems quite similar to the sort of things that MIT have demonstrated some success with last year (http://people.csail.mit.edu/mrub/vidmag/).   Their code is written in Matlib, which is a commercial package, so not much use to me, so I am looking at doing something similar in OpenCV.

But first things first, I need to get OpenCV working.  I am going to use plain old C, because I know the syntax (no funny '<'s in the code that you seem to get in C++).  I may move to Python if I start to need to plot graphs to understand what is happening, so I can use the matplotlib graphical library.

I am using CMake to sort out the make file.  I really don't know how this works - I must have found a tutorial somewhere that told me to create a file called CMakeLists.txt.  Mine looks like:

cmake_minimum_required(VERSION 2.8)

PROJECT( sd )

FIND_PACKAGE( OpenCV REQUIRED )

ADD_EXECUTABLE( sd Seizure_Detector.c )

TARGET_LINK_LIBRARIES( sd ${OpenCV_LIBS} )

Running 'cmake' creates a standard Makefile, and then typing 'make' will compile Seizure_Detector.c and link it into an executable called 'sd', including the OpenCV libraries.   Seems quite clever.

The program to detect a seizure is going to have to look for changes in a series of images in a certain frequency range (a few Hz I think).   To detect this I will need to collect a series of images, process them, and do some sort of Fourier transform to detect the frequency components.

So to get started, grab an image from the networked camera.  This seems to work:
IplImage *origImg = 0;
char *window1 = "Original";
int main() {
    camera = cvCaptureFromFile("rtsp://192.168.1.18/live_mpeg4.sdp");
    if(camera!=NULL) {
    cvNamedWindow(window1,CV_WINDOW_AUTOSIZE);
    while((origImg=cvQueryFrame(camera)) != NULL) {
      procImg = cvCreateImage(cvGetSize(origImg),8,1);
      cvShowImage(window1,origImg);
    }
}
}

I can also smooth the image, and do some edge detection:

    while((origImg=cvQueryFrame(camera)) != NULL) {
      procImg = cvCreateImage(cvGetSize(origImg),8,1);
      cvCvtColor(origImg,procImg,CV_BGR2GRAY);
      //cvSmooth(procImg, procImg, CV_GAUSSIAN_5x5,9,9,0,0);
      smoothImg = cvCreateImage(cvGetSize(origImg),8,1);
      cvSmooth(procImg, smoothImg, CV_GAUSSIAN,9,9,0,0);
      cvCanny(smoothImg,procImg,0,20,3);
   
      cvShowImage(window1,origImg);
      cvShowImage(window2,procImg);
}

Full code at https://github.com/jones139/arduino-projects/tree/master/seizure_detector/video_version.

I am about to update the code to maintain a set of the most recent 15 images (=1 second of video), so I can do some sort of time series analysis on it to get the frequencies.....