Humans are in the business of consuming bananas, whereas neural nets are in the business of peddling bananas. They don't get to actually use these bananas so they can't gain deeper insight into what's they for.
This is the classic lamb vs. mutton issue. Wealthy land owners who use one set of idioms vs. servants who use a different one. Happens to neural nets on human chassis as well.
That touches on something I've pondered for a while:
Maybe the popular sci-fi image of "artificial brains" with humanoid bodies actually turns out to be more accurate than the technical view that robotics and artificial intelligence are mostly separate fields!
What if, instead of machine learning being a source of features we can maybe later build into robots, it's the other way round? What if direct interaction with the real world through robotic limbs and sensors turns out to be what it takes to get AI to stop hallucinating?
What if, furthermore, having a body turns out to be an essential requirement for AI to develop a sense of self? Not just because of sensory feedback, but simply because it provides an anchor point for identity?
This is the classic lamb vs. mutton issue. Wealthy land owners who use one set of idioms vs. servants who use a different one. Happens to neural nets on human chassis as well.