We Tested Our AI Calorie Estimator on Google's Nutrition5k. Portion Size Is Most of the Error.
455 weighed dishes, one photo each: our AI misses calories by 39.5% on average and overestimates small plates by 1.76×. Give it the true weight and the error drops to 20.5%.

Xiaomeng Li · Creator, NanoRhino
We Tested Our AI Calorie Estimator on Google's Nutrition5k. Portion Size Is Most of the Error.
An engineering note. We ran our production coach on a public dataset of weighed meals to see how far one photo can go.
TL;DR
Nutrition5k is a public dataset from Google Research (CVPR 2021). Each dish was weighed ingredient by ingredient, and the calories come from those weights. So every photo has a real answer.
We ran our production coach on 455 dishes from the official test split. One overhead photo, one message: "had this for lunch."
- Photo only: 39.5% mean error (112 kcal on an average dish of 283 kcal). Only 29% of estimates were within ±20%.
- The estimates were biased high by 39%. Small plates were the problem: under 100 g, the model guessed 1.76× the real weight.
- Large plates were fine: for dishes over ~340 kcal, the median error was 28% and the bias was about zero.
- Photo + the true total weight: 20.5% mean error, bias near 0%. Knowing the weight cut the error in half.
The rest of the error comes from calories per gram, and mostly from oil and dressing that the camera cannot see.
This is a different number from the 3.8% we published in July. That 3.8% is on foods with a knowable answer: a label, a number the user said, a simple food. This post is about the hard case: a mixed plate, one photo, nothing else.
Why we ran this
Our own regression gate uses real user conversations. That is good for catching regressions, but the "truth" for a photo of a home-cooked plate is a range, not a number. Nobody weighed it.
Nutrition5k fixes that. The dishes were made in a cafeteria, each ingredient was weighed on a scale, and calories were computed from USDA values. It is released under CC BY 4.0, so we can run it and publish the results.
How we ran it
- Same code path as production. Each dish was a full coach turn: the same agent, the same model (Claude Sonnet 5.5), the same tools that log a meal for a real user. We read the logged calories from the meal file.
- Cold start. A fresh workspace for each dish. No dish memory, no history, no profile to lean on.
- Zero-shot. We did not train or tune anything on Nutrition5k.
- Dishes: all test-split dishes with at least 30 kcal and an overhead RGB photo. That is 455 dishes. One dish produced no log, so 454 were scored.
- Two arms:
- A: photo only. "had this for lunch."
- B: photo + weight. A random 200 of the same dishes, with "about N g total", where N is the true weight rounded to 10 g.
Arm B is not a product feature. Real users almost never tell us the total weight of a mixed plate. Arm B is a controlled experiment. With the weight known, the model still has to decide what is on the plate, how the weight splits between foods, and how many calories each gram has. So comparing A and B tells us how much of the error comes from portion size and how much comes from calories per gram.
The whole run cost $114 in API usage.
Results
| A: photo only (n=454) | B: photo + true weight (n=199) | |
|---|---|---|
| Mean absolute error / mean true kcal | 39.5% (112 kcal) | 20.5% (58 kcal) |
| Median error | 37.0% | 15.7% |
| Within ±20% | 29% | 58% |
| Mean bias | +39.4% | −0.2% |
| Median weight error | 38.9% | 1.3% |
| Median calories-per-gram error | 15.8% | 16.0% |
On the 199 dishes that ran in both arms, the median error went from 38.2% to 15.7%. 142 of 199 dishes got better.
Look at the last two rows. Calories-per-gram error is about 16% in both arms. Giving the weight did not change it. The weight error went from 39% to 1%. Almost all of the improvement is the portion.
In arm A, we also split the variance of the log error into a weight part and a calories-per-gram part: weight is 57% and calories per gram is 43%. Weight is the larger part. It is also the only part with a strong bias in one direction.
Finding 1: the model pulls every portion toward "a normal meal"
When we plot estimated weight against true weight (on a log scale), the slope is 0.73, not 1. Small portions are guessed too big and big portions are guessed too small. The two lines cross at about 470 g.
| True weight | Estimated / true (median) | Dishes |
|---|---|---|
| under 100 g | 1.76× | 107 |
| 100–200 g | 1.44× | 136 |
| 200–350 g | 1.31× | 142 |
| over 350 g | 0.86× | 69 |
Our reading: the model has a prior for "one serving." When it cannot judge size from the photo, it moves toward that prior.
This matters a lot on Nutrition5k because the plates are small. The median dish is 185 g. Many are sample plates from a cafeteria line, not full meals. That is exactly where the over-estimate is largest.
By calories, the same pattern:
| True calories | Median error | Bias |
|---|---|---|
| 30–146 kcal | 48.6% | +85.3% |
| 147–340 kcal | 39.9% | +36.0% |
| 343–1050 kcal | 28.4% | −2.9% |
For a plate the size of a real lunch or dinner, the photo-only estimate was not biased. It was still noisy (28% median error), but it was not high on average.
When we gave the true weight (arm B), the small-plate bias dropped from +85% to +11%, and the median error in every size group was 15–19%.
Finding 2: the oil you cannot see
Calories per gram were close to unbiased for dishes without added fat (median ratio 0.99×, 225 dishes).
For dishes whose weighed ingredients include oil, butter, dressing, mayo or pesto, the model's calories per gram were 0.89× the truth (229 dishes). The model listed a fat item for only 58 of those 229. Oil soaks into food. In a photo it looks like nothing.
This is the main source of the 16% calories-per-gram error. The model is under-counting what it cannot see, not mistaking one food for another.
Finding 3: some of the error is in the dataset
We looked at the 12 worst dishes by hand. In about 4 of them, the label is clearly missing food that is in the photo. For example, the photo shows pizza or a Caesar salad, and the label lists a few grams of broccoli. In 2 others, the label includes things the camera cannot see, like 45 g of olive oil or chicken hidden under other food. A few almond weights also look too high for what is on the plate. Only 3–4 of the 12 are clear portion mistakes by our model.
The worst 10% of dishes carry 29% of the total absolute error. So the average is inflated somewhat by label noise. But the two patterns above (small plates guessed too big, hidden oil missed) are ours. They hold across hundreds of dishes, not just the outliers.
How this compares
The Nutrition5k paper reports its own baselines (Table 3). They are not the same setup as ours, so read this as context, not a ranking:
| Method | Calorie error (MAE / mean) |
|---|---|
| Always guess the dataset mean | 60.2% |
| 2D model trained on Nutrition5k, single photo | 26.1% |
| Model with depth camera | 16.5–18.8% |
| Trained model, true weight given, predicts calories per gram | 9.5% |
| Our coach, photo only, zero-shot | 39.5% |
| Our coach, photo + true weight, zero-shot | 20.5% |
The differences: their models were trained on the same cafeteria, the same camera rig and the same kinds of dishes. Ours has never seen this data. We also dropped dishes under 30 kcal and dishes without an overhead photo. A specialized model trained in-distribution should win on its own test set, and it does.
One more number from the paper: when people estimated the weight of food by eye, dietitians were off by 41% and non-experts by 53% (a small test on 10 plates). Our model's median weight error was 39%. Judging weight from a photo is hard for people too.
What this means for the product
A better vision model would help a little. Better information about the portion helps much more. In this test, knowing the weight halved the error and removed the bias. So the coach is built to get portion information wherever it can:
- Labels and stated numbers win. If you send a nutrition label or say "it's 350 calories," that number is logged as-is. There is no photo estimate to be wrong.
- Ask about the portion when it was a guess. When the portion was assumed and your history says you often eat part of a dish, the coach asks one short question: "all of it, or part?" Small portions are where photo estimates go wrong the most.
- Remember your usual dishes. If you log the same breakfast often, the coach keeps its calories per gram and your usual amount. A label or a correction from you becomes the reference next time, instead of a new guess from a new photo. For you as a user, the practical advice is short: if a plate is small or the food was cooked in oil, add a few words. "Half a bowl," "small plate," "fried in butter." For cooked dishes, the fat is often the biggest item the photo cannot show: "cooked in a tablespoon of olive oil" is about 120 kcal. That one line does more for accuracy than a better photo.
Limits
- Cafeteria food, not home food. Nutrition5k dishes are cafeteria food, shot from directly above with a fixed camera rig. Real user photos are taken at an angle, in different light, often with packaging or a hand for scale.
- Plate sizes are not typical. The median dish is 185 g. Real meals are usually larger, where our bias was close to zero. We cannot claim a single "real-world" error number from this test.
- One model, one run per dish. We did not test other models or repeat runs to measure spread.
- The label noise goes both ways. Some dishes look worse than they are because the label is missing food. We did not remove those dishes; all 454 scored dishes are in the numbers above.
Data and credit
Nutrition5k: Thames et al., "Nutrition5k: Towards Automatic Nutritional Understanding of Generic Food," CVPR 2021 (arXiv 2103.03375). Dataset released by Google Research under CC BY 4.0.
This is an engineering report from the team that builds NanoRhino. It is not medical advice. I am not a clinician.
Try it
Send a photo of your meal and add one line about the portion. Start messaging your coach on WhatsApp or SMS. No app, no signup. Or start with your numbers on our free Accurate TDEE Calculator.
Built with care by Link Heart Limited in Houston, Texas.