It is not only out of guilt. In sports or music, many people are ashamed of being white. In the U.S. it is nearly a foregone conclusion that you will not be a good musician or athlete if you are white. White kids who desire to pursue those things definitely can feel shame which has no feeling of guilt attached to it. I'm not trying to make some grand statement about racism or the effects, just giving a pretty cut and dry counter point to the idea that white people never feel shame about their skin color aside from guilt.
I think the notion that "no one ever thought being white was something to be ashamed of [for reason other than guilt]" does not stem from observaton or experience as much as it is an ideological construct - ie. it is needed to fit nicely into a preassumed political theory