How do AI photo calorie counters work?
Short answer
AI photo calorie counters work in steps. They find the food in the picture, identify each item, estimate how much of it there is, and multiply that amount by nutrition values per gram from a food database. Recognizing the food is the easier part. In Google's Nutrition5k study, most of the error came from judging the amount, and depth data helped with that.
On this page
What happens after you take the photo?
The photo goes through a pipeline of separate jobs. A 2022 systematic review of 78 studies of image-based food recognition describes the usual phases: separating each food from the plate, classifying it as a particular food, and estimating its volume, calories or nutrients.
- Find the food. The system separates each item from the plate and background, a step called segmentation.
- Identify it. Each item is matched to a food, such as rice, grilled chicken or broccoli.
- Estimate the amount, as a volume or a weight.
- Look up the nutrition. Per-gram values from a food database are multiplied by the amount.
- Add it up into totals for the meal.
The methods have changed a lot since 2015. A 2024 scoping review found that systems built before then used hand-crafted algorithms for each step. Deep learning has since largely taken over, and newer multitask models can handle several steps at once.
How does the AI judge portion size?
From visual clues, from objects of known size, and sometimes from depth. It's the hardest step. In Google's Nutrition5k study, a model trained on about 5,000 weighed cafeteria dishes judged calories per gram with 9.5 percent error. Asked for the total calories from a single photo, its error rose to 26.1 percent, nearly three times higher.
Depth data helps. The Nutrition5k team used an overhead depth camera to estimate the volume of food on each plate. Feeding that volume estimate into a separate weight model, combined with the per-gram model, cut the calorie error to 16.5 percent.
Objects of known size help too. The researchers gave people in their survey reference weights, such as a U.S. quarter at 5.6 grams, and a 2025 study told general-purpose AI models to use visible cutlery and plates as size references.
Where do the nutrition numbers come from?
From a food database, once the system knows what the food is and how much there is. Nutrition5k calculated its true values from the USDA's food and nutrient database, multiplying per-gram values by the weight of each item.
This is why the amount matters so much. If the model mistakes one grain for another with similar calories per gram, the total barely moves, but if it misjudges the amount by a third, the total moves by a third.
Where do AI photo counters go wrong?
Mostly on amounts, and on things a camera can't see.
| Step | What can go wrong | How you can help |
|---|---|---|
| Finding and naming foods | Similar-looking foods; mixed dishes that hide ingredients | Check the list of foods it identified |
| Estimating amounts | The largest source of error, especially for big portions | Check the portion sizes it assumed |
| Hidden ingredients | Oil, butter, sugar and sauce don't show in a photo | Add them yourself |
| Database lookup | A close but different entry, such as fried instead of baked | Pick the best match |
Big plates are a particular weak spot. In the 2025 study, three general-purpose AI models estimated 52 standardized food photos. The two better models missed energy by about 36 percent on average, and all three underestimated more as portions grew.
Common questions
Can a general-purpose AI chatbot count calories from a photo?
Roughly. In the 2025 study, the better general-purpose models missed energy by about 36 percent on average and tended to underestimate large portions, which is why the authors judged them not yet suitable where precise numbers matter.
Does a depth sensor make a big difference?
In research, yes. Adding a depth-based volume estimate cut Nutrition5k's calorie error from 26.1 percent to 16.5 percent, mostly by improving the estimate of how much food was on the plate.
Why does the estimate change when I retake the photo?
Because portion size is estimated from what the camera sees. A different angle, distance or light changes the visual clues, so check the amounts rather than the food names first.
Sources
- Dalakleidi KV, Papadelli M, Kapolos I, Papadimitriou K. Applying image-based food-recognition systems on dietary assessment: a systematic review. Advances in Nutrition. 2022;13(6):2590–2619. pubmed.ncbi.nlm.nih.gov
- Chotwanvirat P, Prachansuwan A, Sridonpai P, Kriengsinyos W. Advancements in using AI for dietary assessment based on food images: scoping review. Journal of Medical Internet Research. 2024;26:e51432. pubmed.ncbi.nlm.nih.gov
- Thames Q, Karpur A, Norris W, et al. Nutrition5k: Towards Automatic Nutritional Understanding of Generic Food. CVPR. 2021. openaccess.thecvf.com
- Fridolfsson J, Sjöberg E, Thiwång M, Pettersson S. Performance evaluation of 3 large language models for nutritional content estimation from food images. Current Developments in Nutrition. 2025;9(10):107556. pubmed.ncbi.nlm.nih.gov
Estimates, not medical advice. If you're pregnant, have a medical condition or have a history of disordered eating, talk to a clinician before changing how you eat.
Written by the SeeMeEat editorial team under a pen name.