To understand visual fog computing, it is important to first understand fog computing.
Let’s take a look at how the physical world connects to the cloud. Think of the physical world at the bottom of a pyramid, and the cloud as the top. At the bottom are the edge devices, from industrial temperature sensors to machine vision cameras to baby monitors to license plate readers.
For this article, we’ll concentrate on cameras. A given installation may have one or two or even thousands of cameras. According to the New York Daily News, in 2015 Manhattan had close to 20,000 cameras, and London about 500,000.
All these cameras produce prodigious amounts of data, which presents significant challenges. Handling all that data — transmitting it and processing it — takes computation at three levels: The Edge, Fog, and Cloud.
The cameras connect to local controllers — gateways — where edge computing is done: initial processing, some data display, and some data storage. The data is then passed to the next layer, a bank of servers or fog nodes, which do further processing and send the results to the cloud, where data is analyzed and stored.
Figuring out where to do what processing is far from simple. A presentation by Intel’s Shao-Wen Yang and Wendy Chin at Embedded World 2018, Feb 27-March 1, 2018, Nuremberg, Germany discussed how the computational load can be handled, paying particular attention to offloading and scheduling. The place where you’ll find more cameras and other sensors than almost anywhere is in modern retail. The retail domain faces three significant challenges, according to Chin: customer engagement, as online shopping reduces walk-in traffic; increasing labor costs; and the main one is what she calls “inventory distortion,” which is loosely defined as the cost of understocks plus overstocks, which comes to about $1.13 billion per year. Chin gives an example of a customer being unable to find the desired item and leaving empty-handed.
Efforts to deal with this, continues Chin, include the concept of the responsive retail store, which uses sensors such as RFID, cameras and more so that all items are tracked from the supply chain, to the store, and onto the shelf. Cameras also watch where customers go into the store, and create a hit map of the customers. All this, she explains, is part of the 360º view of the business, from supply chain to customer. The technology that makes this possible is the integration of different kinds of sensors, which includes large numbers of cameras. So responsive retail helps with real-time inventory accuracy, to keep track of misplaced items, and customer concentration, which consists of emotion, or interest, or the amount of attention customers devote to particular items, and how they respond to them.
But, she goes on, this is just the beginning of the responsive retail store. Further along the continuum are staff-free stores, in which the shopper’s face is recognized upon entry, the shopper is greeted automatically, and a kiosk offers items based on known preferences. The customer picks up the desired item — and cameras keep track of everything the customer picks up or puts back, combined with (or aided by) RFID tracking of each item. Payment is made through facial recognition. All this depends on the camera. But what if there are hundreds of cameras in one store, all tracking everything that goes on? That will generate a massive amount of data.
This, says Shao-Wen Yang, is where visual fog computing comes in. Looking again at the pyramid, while smart cameras generate data that goes to video gateways and finally to the cloud, this leaves out the client’s need for metadata and other feedback. Because there is not enough bandwidth to simply send everything to the cloud, the task must be divided among the three layers: edge, gateways, and cloud. Figuring out how to partition all this is the first challenge, and figuring out how to do it takes us into the depths of graph theory. The constraints are device computation capacity and device transmission capacity constraint, Yang explains, where task dependency and device connectivity are directed acyclic graphs, with four edge devices, two intermediate gateways, and the cloud. Partitioning this into 12 interdependent pieces isn’t hard to handle, computationally. But what happens when there are many cameras? With just 16 cameras it requires more than 1000 seconds to define an optimal schedule. And what about the cameras in New York or London? Intel has been working hard on a solution, he says, and has managed to get even these examples down to manageable levels.
The real world is dynamic: cameras are added and subtracted all the time, making workload migration the second challenge — and never forgetting the constraints in networking and communication bandwidth.
The third challenge is universal for all network-connected and Internet activity: security and privacy. And, says Yang, “if we don’t deal with the problem, then visual fog computing will never take off.”
