Click here to Skip to main content
13,409,878 members (37,968 online)
Click here to Skip to main content
Add your own
alternative version

Stats

2.8K views
1 bookmarked
Posted 15 Dec 2017

Edge Data Processing POC with the Up Squared Board

, 15 Dec 2017
This proof of concept (POC) will explore edge computing in a digital signage scenario where the display is recording the number of people viewing the display throughout the day.

Editorial Note

This article is in the Product Showcase section for our sponsors at CodeProject. These articles are intended to provide you with information on products and services that we consider useful and of value to developers.

Intro

With the amount of continuously generated data on the rise, the cost to upload and store that data in the cloud is increasing. Data is being gathered faster than it is stored and immediate action is often required. Sending all the data to the cloud can result in latency and presents risks when internet connectivity is intermittent. Edge computing involves processing data locally for immediate analysis and decisions. Hence it can help reduce network load by sending only critical, processed data to the cloud. This proof of concept (POC) will explore edge computing in a digital signage scenario where the display is recording the number of people viewing the display throughout the day. OpenCV will be used to search the camera feed for faces and Dash* by Plotly* to graph out the results in real time without the cloud.

Up Squared* Board

Figure 1: Up Squared* Board

Set-up

Hardware

The POC uses the Up Squared* board, a high performance maker board with an Apollo Lake processor, with Ubuntu* 16.04 LTS as the OS. For more information on the Up Squared board, please visit: http://www.up-board.org/upsquared/.

The camera used is the Logitech* C270 Webcam with HD 720P which connects to the board by USB.

Software

OpenCV needs to be installed on the Up Squared board to process the images. Reference here for instructions on how to install it on Ubuntu 16.04.

The processed data will be graphed with Dash by Plotly to create a web application in Python*. For more information on Dash: https://plot.ly/dash/

To install Plotly, Dash, and the needed dependencies:

pip install dash==0.19.0  # The core dash backend
pip install dash-renderer==0.11.1  # The dash front-end
pip install dash-html-components==0.8.0  # HTML components
pip install dash-core-components==0.14.0  # Supercharged components
pip install plotly --upgrade  # Plotly graphing library used in examples

#dependencies
pip install pandas
pip install flask
apt get install squlite3 libsqlite3-dev

Face Detection

Face detection is done in Python with OpenCV using the Haar Cascades front face alt algorithm. Each found face is individually tracked in the frame so that each viewer can be counted. This data is recorded to a sqlite database with their face ID and timestamps. Possible expansions to this would be doing facial recognition or using other algorithms to detect demographics of viewers.

The code, seen below, will create and connect to sqlite3 to initialize the database faces.db. The tables inside the database can only be created the first time the code is run, hence it is inside a try clause. OpenCV will then connect to the camera and loop over the feed looking for faces. When it finds a face or faces, it will write send the array of face rectangles to the face_tracking method which will which give each face an ID and then track it based on position. Each face class object stores its current x and y location as well as the time it was last seen and its face ID. If the face disappears for more than 2 seconds, it is aged out of the array and assumed the person moved on or looked away for too long; this will also help with any in-between frames where OpenCV might fail to detect the face where there really is one. This data is then written to the database as the face ID and the timestamp it was seen and to another table is inputted the total number of viewer faces at that timestamp.

Data in the visitorFace table with time stamps and ID

Figure 2: Data in the visitorFace table with time stamps and ID
#!/usr/bin/env python

from __future__ import print_function

# Import Python modules
import numpy as np
import cv2
import Face
import sys
import os
import pandas as pd
import sqlite3
from datetime import datetime
import math

#array to store visitor faces in
facesArray = []

#connect to sqlite database
conn = sqlite3.connect('/home/upsquare/Desktop/dash-wind-streaming-master/Data/faces.db')
c = conn.cursor()

# Create tables
try:
    c.execute('''CREATE TABLE faces
                (date timestamp, time timestamp, date_time timestamp, numFaces INT)''')
    c.execute('''CREATE TABLE visitorFace
                (date_time timestamp, faceID INT)''')
except:
	print('tables already exist')

#find the last visitor faceID to start counting from
df = pd.read_sql_query('select MAX(faceID) FROM visitorFace', conn)
lastFid = df.iloc[0]['MAX(faceID)']
if lastFid is None:
	fid=1
else:
	fid = 1 + int(df.iloc[0]['MAX(faceID)'])

try:
    # Checks to see if OpenCV can be found
    ocv = os.getenv("OPENCV_DIR")
    print(ocv)
except KeyError:
    print('Cannot find OpenCV')


# Setup Classifiers
face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')

#draw rectangles around faces
def draw_detections(img, rects, thickness = 2):
    for x, y, w, h in rects:
        pad_w, pad_h = int(0.15*w), int(0.05*h)
        cv2.rectangle(img, (x, y), (x+w, y+h), (0, 255, 0), thickness)

#track visitor faces based on their location
def face_tracking(rects):
	global fid
	global facesArray
	numFaces= len(rects)
	#age faces out when no faces are detected
	if len(rects) == 0:
		for index, f in enumerate(facesArray):
			if((datetime.now()-f.getLastSeen()).seconds > 2):
				print("[INFO] face removed " + str(f.getId()))
				facesArray.pop(index)
	for x, y, w, h in rects:
		new = True
		xCenter = x + w/2
           	yCenter = y + h/2
		for index, f in enumerate(facesArray):
			#age the face out
			if((datetime.now()-f.getLastSeen()).seconds > 2):
				print("[INFO] face removed " + str(f.getId()))
				facesArray.pop(index)
				new = False
				break
			dist = math.sqrt((xCenter - f.getX())**2 + (yCenter - f.getY())**2)
			#found an update to existing face
			if dist <= w/4 and dist <= h/4:
				new = False
				f.updateCoords(xCenter,yCenter,datetime.now())
				c.execute("INSERT INTO visitorFace VALUES (strftime('%Y-%m-%d %H:%M:%f'),"+ str(f.getId())+")")	
				break		
		#add a new face	
		if new == True:
			print("[INFO] new face " + str(fid))
			f = Face.Face(fid, xCenter, yCenter, datetime.now())
			facesArray.append(f)
			fid += 1
			c.execute("INSERT INTO visitorFace VALUES (strftime('%Y-%m-%d %H:%M:%f'),"+ str(f.getId()) +")")	
	print(len(facesArray))
	c.execute("INSERT INTO faces VALUES (DATE('now'),strftime('%H:%M:%f'),strftime('%Y-%m-%d %H:%M:%f'), "+ str(len(facesArray))+")")

try:
    # Initialize Default Camera
    webcam = cv2.VideoCapture(0)
    # Check if Camera initialized correctly
    success = webcam.isOpened()
    if success == True:
        print('Grabbing Camera ..')
    elif success == False:
        print('Error: Camera could not be opened')

    while(True):
        # Read each frame in video stream
        ret, frame = webcam.read()
        # Perform operations on the frame here
        # First convert to Grayscale 
        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
        # Next run filters
        gray = cv2.equalizeHist(gray)
        faces = face_cascade.detectMultiScale(gray, 1.3, 5, minSize=(30, 30), flags=cv2.CASCADE_SCALE_IMAGE)
        print('Number of faces detected: ' + str(len(faces)))
	#send the faces to be tracked for visitorFace faces
	face_tracking(faces)

	#draw on the cv viewing window
        out = frame.copy()
	draw_detections(out,faces)
        cv2.imshow('Facetracker', out)

	#commit the sqlite insertions to the database
        conn.commit()

        # Wait for Esc Key to quit
        if cv2.waitKey(5) == 27:
            break
    # Release all resources used
    webcam.release()
    cv2.destroyAllWindows()
    
except cv2.error as e:
    print('Please correct OpenCV Error')
Code 1: faceCounting.py code for detecting faces

Below is the Face class for each visitor face in the facesArray.

class Face:
    path = []
    def __init__(self, id, x, y, lastSeen):
        self.id = id
        self.x = x
        self.y = y
        self.lastSeen = lastSeen	
    def getId(self):
        return self.id
    def getX(self):
        return self.x
    def getY(self):
        return self.y
    def getLastSeen(self):
        return self.lastSeen
    def updateCoords(self, newX, newY, newLastSeen):
        self.x = newX
        self.y = newY
	self.lastSeen = newLastSeen
Code 2: face.py for Face Class

Processing and Graphing

To visualize the data after doing some data processing on it, Dash will be used to create a graph that will update itself as new data comes in. The graph will also be in a web application running at localhost:8050. This way insights and information can be seen directly at the edge.

In addition to graphing the number of faces data, the data will be smoothed using Pandas exponentially weighted moving average (EWMA) method to create a less jagged line.

#query the datetime and the number of faces seen at that time
df = pd.read_sql_query('SELECT numFaces, date_time from faces;'
                        , con)
#do a EWMA mean to smooth the data
df['EWMA']= df['numFaces'].tail(tail+100).ewm(span=200).mean()
Code 3: Code snippet to query the number of faces and do EWMA smoothing

Face Counting Data graph and EWMA smoothed data

Figure 3: Face Counting Data graph and EWMA smoothed data

From the visitorFace table, the face IDs and timestamps will be used to calculate the time someone looked at the display, sometimes referred to as dwelling time. This type of data could be monetized to drive revenue of ads at peak traffic times and report ad impact/reach. The dwelling time is calculated by getting the first and last timestamp for each face ID and then finding the difference between them.

#query the datetime and the number of faces seen at that time, we need this for the next query
    dfFaces = pd.read_sql_query('SELECT numFaces, date_time from faces;'
                            , con)
    #calculate the viewing time of each visitor face
    #query the first date time and last date time that a visitor face was seen from the tail of the dfFaces data
    #taking the data from the tail will make the Viewing Time Data graph and the Face Counting Data Graph line up
    df = pd.read_sql_query("""select faceID, MIN(date_time) as First_Seen , MAX(date_time) as Last_Seen 
		from (select * from visitorFace Where date_time > "%s") group by faceID;""" %(min(dfFaces['date_time'].tail(tail)))
		, con)
    #calculate the seconds a visitor face viewed the display
    df['VD']= (pd.to_datetime(df['Last_Seen']) - pd.to_datetime(df['First_Seen'])).dt.seconds
Code 4: Code snippet to query the viewing time for each visitor face

Scatter plot of each visitor face viewing time

Figure 4: Scatter plot of each visitor face viewing time

The Dash graph application itself has a few key pieces: the page layout, the callback for the update interval, and the graph components. The layout for this application has four main graphs with four headers: Face Counting Data, Viewing Time Data, Daily Face Count, and Hourly Face Count. The interval will call the callback every second to pull the last 1000 data entries from the sqlite database to graph, hence the graph will update itself as new data is added to database. For the graph components, each line or scatter dot is called a trace and the overall graph’s axis’s values are defined.

from __future__ import print_function
import dash
import dash_core_components as dcc
import dash_html_components as html
from dash.dependencies import Input, Output, State, Event
import plotly.plotly as py
from plotly.graph_objs import *
from flask import Flask
import numpy as np
import pandas as pd
import os
import sqlite3
import datetime as dt
from datetime import timedelta

#number of most recent datapoints to grab
tail = 1000

app = dash.Dash()

#dash page layout
app.layout = html.Div([
    html.Div([
        html.H2("Face Counting Data")
    ]),
    html.Div([
        html.Div([
            dcc.Graph(id='num-faces'),
        ]),
        dcc.Interval(id='face-count-update', interval=1000),
    ]),
    html.Div([
        html.H2("Viewing Time Data")
    ]),
    html.Div([
        html.Div([
            dcc.Graph(id='viewing-data'),
        ]),
        dcc.Interval(id='viewing-data-update', interval=1000),
    ]),
    html.Div([
        html.H2("Daily and Hourly Face Count")
    ]),
    html.Div([
        html.Div([
            dcc.Graph(id='num-faces-daily'),
        ]),
        dcc.Interval(id='face-count-daily-update', interval=1000),
    ]),
    html.Div([
        html.H2("Hourly Face Count")
    ]),
    html.Div([
        html.Div([
            dcc.Graph(id='num-faces-hourly'),
        ]),
        dcc.Interval(id='face-count-hourly-update', interval=1000),
    ]),
], style={'padding': '0px 10px 15px 10px',
          'marginLeft': 'auto', 'marginRight': 'auto', "width": "1200px",
          'boxShadow': '0px 0px 5px 5px rgba(204,204,204,0.4)'})



#callback method for face counting data graph interval
@app.callback(Output('num-faces', 'figure'), [],
              [],
              [Event('face-count-update', 'interval')])
def gen_num_faces():
    con = sqlite3.connect("./Data/faces.db")
    #query the datetime and the number of faces seen at that time
    df = pd.read_sql_query('SELECT numFaces, date_time from faces;'
                            , con)
    #do a EWMA mean to smooth the data
    df['EWMA']= df['numFaces'].tail(tail+100).ewm(span=200).mean()

    trace = Scatter(
	x=df['date_time'].tail(tail),
        y=df['numFaces'].tail(tail),
        line=Line(
            color='#42C4F7'
        ),
        hoverinfo='x+y',
        mode='lines',
        showlegend=False,
        name= '# Faces'
    )

    trace2 = Scatter(
	x=df['date_time'].tail(tail),
        y=df['EWMA'].tail(tail),
        line=Line(
            color='#FF4500'
        ),
        hoverinfo='x+y',
        mode='lines',
        showlegend=False,
        name= 'EWMA'
    )

    layout = Layout(
        height=600,
        xaxis=dict(
            range=[min(df['date_time'].tail(tail)),
                   max(df['date_time'].tail(tail))],
            showgrid=True,
            showline=True,
            zeroline=False,
            fixedrange=True,
            nticks=max(8,10),
            title='Date Time'
        ),
        yaxis=dict(
            range=[0,
                   max(df['numFaces'].tail(tail))],
            showline=True,
            fixedrange=True,
            zeroline=False,
            nticks=max(2,max(df['numFaces'].tail(tail))),
	    title='Faces'
        ),
        margin=Margin(
            t=45,
            l=50,
            r=50,
            b=100
        )
    )

    return Figure(data=[trace,trace2], layout=layout)


#callback method for viewing time data graph interval
@app.callback(Output('viewing-data', 'figure'), [],
              [],
              [Event('viewing-data-update', 'interval')])
def gen_viewing_data():
    con = sqlite3.connect("./Data/faces.db")
    #query the datetime and the number of faces seen at that time, we need this for the next query
    dfFaces = pd.read_sql_query('SELECT numFaces, date_time from faces;'
                            , con)
    #calculate the viewing time of each visitor face
    #query the first date time and last date time that a visitor face was seen from the tail of the dfFaces data
    #taking the data from the tail will make the Viewing Time Data graph and the Face Counting Data Graph line up
    df = pd.read_sql_query("""select faceID, MIN(date_time) as First_Seen , MAX(date_time) as Last_Seen 
		from (select * from visitorFace Where date_time > "%s") group by faceID;""" %(min(dfFaces['date_time'].tail(tail)))
		, con)
    #calculate the seconds a visitor face viewed the display
    df['VD']= (pd.to_datetime(df['Last_Seen']) - pd.to_datetime(df['First_Seen'])).dt.seconds

    #declare the traces array to hold a trace for each visitor face
    traces = []
    for i in range(len(df.index)):
        traces.append(Scatter(
            x= [df['First_Seen'][i],df['Last_Seen'][i]],
            y= [i],           
            hoverinfo='name',
            mode='markers',
            opacity=0.7,
            marker={
                'size':df['VD'][i],
                'line': {'width':0.5,'color':'white'}
            },
            showlegend=False,
            name= df['VD'][i]
        ))
    layout = Layout(
        height=600,
        xaxis=dict(
            range=[min(dfFaces['date_time'].tail(tail)),
                   max(dfFaces['date_time'].tail(tail))],
            showgrid=True,
            showline=True,
            zeroline=False,
            fixedrange=True,
            nticks=max(8,10),
            title='Date Time'
        ),
        yaxis=dict(
            range=[0,len(df.index)],
            showline=True,
            fixedrange=True,
            zeroline=False,
            nticks=max(2,len(df.index)/4),
	    title='Viewer'
        ),
        margin=Margin(
            t=45,
            l=50,
            r=50,
            b=100
        )
    )

    return Figure(data=traces, layout=layout)


#callback method for daily face count graph interval
@app.callback(Output('num-faces-daily', 'figure'), [],
              [],
              [Event('face-count-daily-update', 'interval')])
def gen_num_faces_daily():
    weekAgo = dt.datetime.today() - timedelta(days=6)

    con = sqlite3.connect("./Data/faces.db")
    #query the first date time and last date time that a visitor face was seen
    df = pd.read_sql_query("""select faceID, MIN(date_time) as First_Seen , MAX(date_time) as Last_Seen 
		from visitorFace where date_time >= "%s" group by faceID;""" %weekAgo , con)
    #reset the index to do a groupby
    df= df.set_index(['First_Seen'])
    df.index = pd.to_datetime(df.index)
    #group the visitor faces by day and record the number of faces per day
    dfDaily = df.groupby(pd.TimeGrouper('D')).size().reset_index(name='counts')

    trace = Scatter(
	x=dfDaily['First_Seen'],
        y=dfDaily['counts'],
        line=Line(
            color='#42C4F7'
        ),
        hoverinfo='x+y',
        mode='lines',
        name= 'Daily'
    )

    layout = Layout(
        height=600,
        xaxis=dict(
            range=[min(dfDaily['First_Seen']),
                   max(dfDaily['First_Seen'])],
            showgrid=True,
            showline=True,
            zeroline=False,
            fixedrange=True,
            nticks=max(2,len(dfDaily.index)),
            title='Date Time'
        ),
        yaxis=dict(
            range=[min(0, min(dfDaily['counts'])),
                   max(dfDaily['counts'])],
            showline=True,
            fixedrange=True,
            zeroline=False,
            nticks=max(4,len(dfDaily.index)),
	    title='Faces'
        ),
        margin=Margin(
            t=45,
            l=50,
            r=50,
            b=100
        )
    )

    return Figure(data=[trace], layout=layout)

#callback method for hourly face count graph interval
@app.callback(Output('num-faces-hourly', 'figure'), [],
              [],
              [Event('face-count-hourly-update', 'interval')])
def gen_num_faces_hourly():
    weekAgo = dt.datetime.today() - timedelta(days=6)
    con = sqlite3.connect("./Data/faces.db")
    #query the first date time and last date time that a visitor face was seen
    df = pd.read_sql_query("""select faceID, MIN(date_time) as First_Seen , MAX(date_time) as Last_Seen 
		from visitorFace where date_time >= "%s" group by faceID;""" %weekAgo , con)
    #reset the index to do a groupby
    df= df.set_index(['First_Seen'])
    df.index = pd.to_datetime(df.index)
    #group the visitor faces by day and record the number of faces per hour
    dfHourly = df.groupby(pd.TimeGrouper('H')).size().reset_index(name='counts')


    trace = Scatter(
	x=dfHourly['First_Seen'],
        y=dfHourly['counts'],
        line=Line(
            color='#42C4F7'
        ),
        hoverinfo='x+y',
        mode='lines'
    )

    layout = Layout(
        height=600,
        xaxis=dict(
            range=[min(dfHourly['First_Seen']),
                   max(dfHourly['First_Seen'])],
            showgrid=True,
            showline=True,
            zeroline=False,
            fixedrange=True,
            nticks=14,
            title='Date Time'
        ),
        yaxis=dict(
            range=[min(0, min(dfHourly['counts'])),
                   max(dfHourly['counts'])],
            showline=True,
            fixedrange=True,
            zeroline=False,
            nticks=max(2,max(dfHourly['counts'])/10),
	    title='Faces'
        ),
        margin=Margin(
            t=45,
            l=50,
            r=50,
            b=100
        )
    )

    return Figure(data=[trace], layout=layout)

external_css = ["https://cdnjs.cloudflare.com/ajax/libs/skeleton/2.0.4/skeleton.min.css",
                "https://fonts.googleapis.com/css?family=Raleway:400,400i,700,700i",
                "https://fonts.googleapis.com/css?family=Product+Sans:400,400i,700,700i"]


for css in external_css:
    app.css.append_css({"external_url": css})

if __name__ == '__main__':
    app.run_server()
Code 5: app.py code for Dash by Plotly web graph

The above graphs give a very close up view of the traffic in front of the display. To get a higher level picture of what is going on, let’s arrange the data and look at how many visitor faces per day and per hour from the last week.

weekAgo = dt.datetime.today() - timedelta(days=6)

    con = sqlite3.connect("./Data/faces.db")
    #query the first date time and last date time that a visitor face was seen
    df = pd.read_sql_query("""select faceID, MIN(date_time) as First_Seen , MAX(date_time) as Last_Seen 
		from visitorFace where date_time >= "%s" group by faceID;""" %weekAgo , con)
    #reset the index to do a groupby
    df= df.set_index(['First_Seen'])
    df.index = pd.to_datetime(df.index)
    #group the visitor faces by day and record the number of faces per day
    dfDaily = df.groupby(pd.TimeGrouper('D')).size().reset_index(name='counts')
   dfHourly = df.groupby(pd.TimeGrouper('H')).size().reset_index(name='counts')
Code 6: Code snippet to query face count daily and hourly

Daily Face Count graph

Figure 5: Daily Face Count graph

Hourly Face Count Graph

Figure 6: Hourly Face Count Graph

Summary

So concludes this POC of gathering and analyzing data at the edge. Running the face tracking application and the Dash application at the same time will show data graphing in real time. From here any of the processed data is ready to send to the cloud to store for long term.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Intel Corporation
United States United States
You may know us for our processors. But we do so much more. Intel invents at the boundaries of technology to make amazing experiences possible for business and society, and for every person on Earth.

Harnessing the capability of the cloud, the ubiquity of the Internet of Things, the latest advances in memory and programmable solutions, and the promise of always-on 5G connectivity, Intel is disrupting industries and solving global challenges. Leading on policy, diversity, inclusion, education and sustainability, we create value for our stockholders, customers and society.
Group type: Organisation

44 members


You may also be interested in...

Pro
Pro

Comments and Discussions

 
-- There are no messages in this forum --
Permalink | Advertise | Privacy | Terms of Use | Mobile
Web02 | 2.8.180221.1 | Last Updated 15 Dec 2017
Article Copyright 2017 by Intel Corporation
Everything else Copyright © CodeProject, 1999-2018
Layout: fixed | fluid