Oh interesting. I wouldn't put it past them that they include that data in the input though. Why are you certain they don't? (I guess that they don't, too, because someone reported getting identical calorie readings from giving photos to Gemini 1.5 directly)
They’re using some off the shelf multimodal LLM API, perhaps Gemini 1.5 or something else. And no off the shelf VLM API is capable of handling 4 channel images (RGB+D).
You can take crude measurements from the depth map and provide as text. Or perhaps provide the depth map as an image next to the actual image as those are certainly in training data. I haven’t tested these
