I think he's internally dissatisfied with his own answer to the question. He's certainly a man who wanted to know more than he could ever hope to achieve. I think things like this irked him at a deep level.
You come up to him. "Hey, so magnets, these trivial things that children play with, can you explain it to me?"
His real answer would be "no. not to my satisfaction, I can't"
That isn't it, though. Feynman explains magnetism perfectly well in his lectures, he's just saying that he can't give a decent math-free answer in a few minutes to an interviewer with no science background. Any answer that short would not functionally be any better than "magnets work because of magic".
You come up to him. "Hey, so magnets, these trivial things that children play with, can you explain it to me?"
His real answer would be "no. not to my satisfaction, I can't"