Machine vision provides an alternative form of data input into IoT systems. Typically it is used whenever there is a requirement for visual inspection and where imagery is available. Images often from surveillance or security systems, but drones and smart phones have enabled a lot more ways to collect imagery.
Common applications include asset inspection, product inspection, traffic counting, and can be used to facilitate decision making in the supervisory systems of many applications involved with smart cities. It may even be used as an input to project management of construction projects or monitoring vegetation/species growth in mine site rehabilitation.
Another possible future application is in the monitoring of offenders if society decides to replace traditional prisons with technological incarceration.
Machine vision usually works by combining images with other applications such as optical character recognition (OCR) to extract text or facial recognition and face detection. Typical analytical techniques include blob analysis and edge detection to find and classify the parts of the image that are of interest.
Machine vision is well-suited to defined, repetitive tasks such as completing inspections and detecting defects. It delivers consist results but is not suitable for tasks with a high level of ambiguity and uncertainty. Machine learning allows the collection and categorisation of data to allow systems to adaptively learn and improve over time by highlighting changes and anomalies in the data.
Machine vision differs from human vision. Human vision is adept at managing decision making, change, variation and filling in mission information when images are viewed in context. However, it is not always precise. The key concepts of human vision, machine vision and machine learning are summarised in the following diagram.
Diagram courtesy of Ryan Messina, Messina Vision Systems
The design process for automating systems using machine vision can be an iterative process. The repetitive tasks are replaced with machine vision first, but still using human vision to address variable conditions and decision-making. This system can be continuously improved and further automated using machine learning and information, and addressing environmental factors, as discussed further in the case studies below and the designing for IoT page.
A key design question is whether to employ edge computing principles or transmit imagery via a network to another location for analysis. For example in the traffic counting case study below it would be better to just send a total count of vehicles periodically rather than transmitting the image file. This will require greater processing capability and power requirements in the device capturing the images, so a tradeoff may be required. One advantage of transmitting data rather than images is that the device can be connected using an LPWAN network. However if the image is transmitted to the cloud then more advanced data analytical techniques such as cognitive computing or machine learning may be employed.
A related design question is whether data needs to be transmitted in real time or not. Some applications such as product inspection need to take immediate action if faults are detected. However, for battery operated IoT devices power is always a primary consideration. Techniques such as only taking a photo when a person, object or animal moves past a camera may be employed to minimise power usage.
Another concept which is becoming more important for automated machine vision systems is 'transfer learning': or the ability to transfer some of the learning of established automated systems to different, but related applications.
Currently, one of the major cost constraints in machine vision systems is the labour required to label the data, and check and validate the system. So this aspect needs to be considered in the business case for machine vision systems.
Finally, machine vision systems can be vulnerable to changes in its environment environment. For example, an automated car could be designed to stop at a traffic light that looks a certain way. If the traffic light design is changed, the car may no longer stop for a red light. A full risk analysis should be conducted in the context of each application.
Machine Vision: Traffic Counting
One example of iterative design of an automated system using machine vision is a traffic counting system designed by Messina Vision Systems.
Before automation, traffic engineers used human vision to count vehicles, and make judgements based on the types of vehicle, incidents and speed in video files that were between 24 hours and one month long. They engaged Messina Vision Services to automate the process with a required 99% accuracy.
The first iteration of the solution involved designing a motion sensor to break down the repetitive task of watching the video of the road or intersection, and the more complicated task of classifying vehicle type, speed and actions. The motion detector took snapshots, or images of sections of the video. It only achieved 90% accuracy, but engineers were able to put the snapshots from the motion sensor in a folder and spend an hour, rather than 24 hours, counting the vehicles to ensure accurate counting of vehicles.
The next iteration of the design addressed the two main flaws in the original system: double counting cars and counting blank spaces where there were no cars. A script was written to analyse the machine-processed video and remove duplicates and empty frames. It achieved 95% accuracy. This used the machine learning concept of clustering.
To increase the accuracy of the system further, the client addressed environmental issues such as birds perching on cameras, large trucks blocking the camera image and sun reflections on the lens. This was done by using quality control measures to position the cameras in such a way that these environmental factors were minimised or eliminated.
Machine Vision: Asset Inspection
A second example is a system designed by Messina Vision Systems to automate pit inspections using machine vision. This was originally a four week manual process: first collecting the data, then converting it to usable information, reporting and reviewing.
The first iteration was a simple system where photographs were taken with a smart phone to track changes, and a form filled out during the manual inspection. This reduced the process to three weeks as usable information was collected on site.
The next iteration was to pre-fill information in the reports with information collected on site using tablets, which reduced the task to two weeks.
In the future, this system could be further automated using robotic inspectors to perform the inspection, and predict when they need to perform them.
The information on this page has been sourced primarily from the following:
A webinar titled 'How Machine Vision Helps Realise the Smart City Concept' by Ryan Messina, Director and System Engineer, Messina Vision Systems.
Edited by Nadine Cranenburgh