An RGB value per pixel is not all of the information you need to render a painting accurately! To capture the effect you describe, paintings should be scanned with something that can capture a BDRF. Then viewing them in VR could give a very accurate experience.
Yes, if you want to go all the way. There are many simplified formulations of BDRFs that are commonly used that ignore polarization and even wavelength dependent effects (e.g. metallic/roughness used in physically based rendering for games which has only two parameters on top of the base color), but they would still be a huge improvement over not capturing any BDRF data at all.
