Patent Overview
Title
Multi-sensor array including an IR camera as part of an automated kitchen assistant system for recognizing and preparing food and related methods
Inventors
- Ryan W. Sinnet 🙋♂️
- Robert Anderson
- Zachary Zweig Vinegar
- William Werst
- David Zito
- Sean Olson
Patent No.
US 10,919,144 B2
Issued
Patent Family
The patent was extended in the following continuation patent(s) (wherein the key differences are described):
Abstract
An automated kitchen assistant system inspects a food preparation area in the kitchen environment using a novel sensor combination. The combination of sensors includes an Infrared (IR) camera that generates IR image data and at least one secondary sensor that generates secondary image data. The IR image data and secondary image data are processed to obtain combined image data. A trained convolutional neural network is employed to automatically compute an output based on the combined image data. The output includes information about the identity and the location of the food item. The output may further be utilized to command a robotic arm, kitchen worker, or otherwise assist in food preparation. Related methods are also described.
The Problem We Solved
In commercial kitchens, it’s extremely challenging for robots to “see” food properly. Traditional cameras struggle to identify food items against similar-colored cooking surfaces, especially as food changes appearance while cooking. Human chefs instinctively use multiple senses, including sight and temperature awareness, but robots traditionally relied only on regular vision.
The Innovation
We developed a system that combines multiple types of sensors – most importantly adding an infrared (IR) camera to traditional vision systems. This multi-sensor approach gives robots a much more reliable way to:
- Identify different food items on a cooking surface
- Track how well-cooked food is becoming
- Precisely locate food items for robot manipulation
- Work effectively in the messy, variable environment of a real kitchen
The system works because food at different cooking stages has distinct temperature patterns that are invisible to normal cameras but clear in infrared imaging. By teaching the computer to combine regular images with thermal information, the system makes much more accurate decisions.
Scope of the Patent
This patent is a landmark patent in the kitchen robotics space as it was one of the first patents filed in this space and makes broad and valuable claims. The patent claims both a method and a system which involves fusing data from an IR camera and a visible light camera and using convolutional neural networks to process the images from the sensor data to determine what objects are present and to track them through space and time. The foundational nature of this invention does not include additional unnecessary frills which is important because it means that this patent covers the most common and effective solutions to the problem it solves.
In a later continuation patent of the same title (US-11618155-B2: Multi-sensor array including an IR camera as part of an automated kitchen assistant system for recognizing and preparing food and related methods), we claimed such a system with two visible light cameras instead of an IR camera, which further provided further coverage beyond this present patent. Note that the fact that IR appears in the title of this later patent does not impact the claims which do not require an IR camera.
Why It Matters
This technology is foundational for practical cooking robots. It enables robotic kitchen assistants to reliably:
- Know when to flip burgers or remove food from heat
- Understand what’s happening on a grill or cooking surface
- Work alongside human kitchen staff safely
- Maintain consistent food quality
This patent represents a key breakthrough that makes kitchen automation practical in real-world settings, potentially transforming how commercial kitchens operate.