Mobile AI pedestrian detection: Sentinel VISION AI

Mike Davis of PRM Engineering discusses the waste sector applications of artificial intelligence enabled hazardous monitoring systems.

Engineers have been developing and trialling self-driving car prototypes for decades, and while we’re not yet “permanent backseat drivers”, the technology is developing rapidly.

As the product of artificial intelligence (AI), a central element of autonomous vehicle research is the ability to detect and avoid pedestrians and other obstacles on the road.

As the machine learning movement progresses, AI in industrial applications is also moving beyond the automotive industry. To that end, PRM Engineering Services, a Brisbane-based control and engineering solutions firm, has introduced the latest in AI human recognition to the waste industry via hazardous monitoring.

Developed over three years, Sentinel VISION AI monitors danger zones in real time to reliably alert machine operators and pedestrians of safety breaches.

According to Mike Davis, PRM Group Managing Director, Sentinel Vision AI was developed in response to growing legislative and cultural safety impetus. As a result of this change, he says companies are becoming more aware of risks and are seeking better solutions.

Mike adds, however, that safety system development has typically focused solely on operators or site employees. What sets Sentinel Vision AI apart, he says, is its dual operator and public monitoring approach, which works to support not just workplace health and safety, but also behavioural change.

“When operating in an essential service such as waste management and resource recovery, it’s not enough to simply install operator-focused hazard systems,” Mike says.

“Kerbside collection truck drivers, for instance, are faced with a number of uncontrollable variables, such as pedestrians or cyclists. Furthermore, operators at public transfer stations are faced with the risk of members of the public getting hurt by moving machinery, arms offloading dustbins or compactors.”

To address these challenges, PRM developed Sentinel Vision AI, which uses cameras and machine learning to recognise when a human enters a hazardous zone.

“A lot of other safety detection systems register all objects, be they wheelie bins, power poles or other vehicles. They don’t actually distinguish if the ‘object’ is a person. In contrast, Sentinel Vision uses AI to create an alert when people are detected,” Mike explains.

“While operators certainly don’t want to hit inanimate objects, the focus needs to be on human safety. And when beeps and buzzers are going off regularly, operators often become desensitised.”

Mike highlights that in a waste context, operators often work in close proximity to other vehicles, buildings, objects and, most importantly, people. As such, alarms go off regularly, causing operators to sometimes ignore or even switch off safety systems.

PRM’s human-only detection system incorporates multiple cameras, which are mounted to mobile machines such as wheel loaders, excavators and refuse trucks. The cameras are pointed at the zone operators want to detect, Mike says, with as many or as few cameras installed as needed.

“The images are then processed by an AI neural network that is trained using machine learning and information input algorithms. The network pulls the image out, and if there is anything that looks like a person, or part of a person, an alarm will sound,” Mike says.

Detection zones are customisable and determined using a drag-and-drop interface, with an additional option of pre-warning zones.

Sentinel Vision uses a combination of visual and audible alarms, which alert the operator and pedestrian that they’re in the wrong place.

In addition to visual and audible alarms, the system has a unique voice alarm system that speaks directly to pedestrians.

“Beeps and buzzers are easy to dismiss, but a human voice speaking from the side of a mobile vehicle is very difficult to ignore,” Mike says.

Sentinel Vision can be retrofit to any existing machine, with optional motion-stop integration providing hard engineered control at closer distances. He adds that with a detection distance up to 12 metres in optimal conditions, the system is more vigilant and reliable than human spotters or traditional electronic tag monitoring systems.

As a value add for the sometimes dusty and unlit environments of waste operations, Mike says the system is enabled to work in low light conditions. He adds that user feedback highlights Sentinel Vision’s ability to detect under harsh conditions as a unique industry standout.

According to the PRM team, Sentinel Vision has been trialled by a number of top tier waste management companies. Mike adds that results have been positive, with multiple companies reporting that when people are alerted by the voice alarm, behavioural change is noticeable.

“By alerting pedestrians as well as operators, we’ve developed a system that supports behavioural change and greater awareness. Ideally, over time, Sentinel Vision will be activated a lot less, because people will have learnt not to walk in front or behind active machines and vehicles.”

Despite Sentinel Vision representing an exciting new development for PRM, it’s one in a long line of engineering and technology innovations for the company.

Mike says PRM is an integrator and developer of a number of products specialising in safety systems, such as height limiters, pedestrian detection and bespoke engineering products.

“We have an extensive history in the earthmoving industry, as well as interfacing systems for OEMs,” he says.

“Our talented team of engineers, together with a support staff of electrical, hydraulic and installation specialists, are able to offer end-to-end innovative and customised solutions to our wide range of customers.”

Related stories:

Rise of the machines: ZenRobotics

Following its success internationally, artificial intelligence designed by ZenRobotics is poised to support the Australian waste sector with efficiencies and higher fraction purity.

The fear of robots taking over society in some dystopian future is a cliché-ridden notion that harks back to the 80s Terminator franchise.

Almost 40 years on from the iconic production, robots by and large still serve as an adjunct, rather than a threat to human existence.

While some modern futurists like Yuval Noah Harari go as far as to suggest human consciousness as we know could change over the century thanks to robots, this reality is far from the contemporary.

For example, futurist Bernard Marr argues critical thinking, creativity, strategy, technology management, installation and upkeep are skills robots can’t do well.

While some can be resistant to change, robots are poised to support the recycling workforce by taking menial tasks off their hands and creating new jobs.

That’s according to Juha Meiskonen, Head of Sales at ZenRobotics. Based in Helsinki, Juha has seen that in many cases, obsolescence of roles like picking empower those workers to take on more challenging tasks such as site management.

“Repetitive tasks are often more suited to a machine where the downsides of being a human could be getting tired, not being focused or being in a dangerous situation,” Juha explains.

Based in Queensland’s southeast, Robots in Waste has been working with ZenRobotics since its 2014 inception. ZenRobotics was founded in 2007 and entered the waste sector around 2010. The company has been most active in Europe since then, but expanded to Asia and North America around 2014-15.

Robots in Waste, which distributes ZenRobotics and other technologies locally, is now looking at accelerating its presence in the Australian market.

In traditional industrial automation, robots operate in defined, structured environments. In waste treatment, the process is less predictable, with complex waste stream compositions and harsh working environments such as temperature changes, dust and dirt.

Artificial intelligence (AI), however, has changed the game. According to ZenRobotics, unlike car manufacturing, waste processing is a chaotic, unstructured environment that is extremely difficult to automate. The company was pleased to take on the challenge, working to develop a robot that could match, if not exceed, human perception.

In 2010, ZenRobotics pioneered its own AI product based on the latest research in the field. In developing the solution, ZenRobotic’s ZENBRAIN hardware was designed to be flexible and adaptive to recognise, grab and sort objects from the waste stream. ZENBRAIN can not only perform complex tasks, but also handle collisions.

Juha says that what is unique about ZenRobotics is the company developed its own machine learning algorithms.

“Robotics in manufacturing requires a homogenous knife, clean environment and we wanted to apply the same efficiency of robotics to an industry which is more chaotic and heterogeneous.”

This, he says, is where the ZenRobotics system was developed to readily identify and recover objects, much like a human can with hand-eye coordination.

Juha says that initial development and testing involved training the robot to recognise new fractions. Now, operators can do this at their own accord, training the robot to recognise fractions in a similar fashion to a human.

Over the years, ZenRobotics expanded to Europe, Asia and North America. While Robots in Waste deployed a ZenRobotics system in Australia in 2017, it is hoping to increase this significantly and has already received extensive interest from a range of companies.

In addition to providing materials recovery facilities with increased efficiency and productivity, the machine can be run 24/7 with constant speed.

Additionally, the sophisticated technology aims to improve the purity of end fractions with sensors and AI software allowing for versatile sorting capabilities. Juha says this may come in handy when end users need to increase their purity to achieve a better price per tonne. End users can train the robot to sort specific objects, not just materials.

AI and digitisation also produces more data on the waste, which may help companies improve and monitor their operations. Robots in Waste’s Jim Duncan says that the digitisation of waste will help drive robotics forward, as the recycling sector moves from a feeling-based operation to a data-driven philosophy.

Two products that have proved popular for ZenRobotics internationally are the fast picker, suitable for municipal solid waste, and the heavy picker, ideal for commercial and industrial and construction and demolition waste.

The heavy picker uses the company’s own robot design with AI software that can be easily upgraded and in-house support guarantees a safe investment. Optional features comprise sorting belt speed control adaptable to the waste stream in addition to a feed rate control for upstream feeding and dosing. On top of replacing manual processes, the heavy picker can replace excavator hours, adjust waste sorting tasks and provide hybrid sorting.

The Fast Picker’s robust and compact design is suitable for demanding environments with an efficient solution for quality control. A single robot arm can simultaneously sort four different fractions to achieve up to 98 per cent purity.

The sensor includes NIR, 3D, hi-res, an imaging metal detector and VIS sensors. With a single sorting bay, the Fast Picker can be retrofitted to existing materials recovery facilities for different conveyor widths and multi-lane conveyors.

Software upgrades will also help future-proof the technology to work with various sensors into the future. With this in mind, Jim remains excited about the prospect of revolutionising the recycling sector with faster and smarter machinery.

Related stories: