I don't know CAD models, but it would probably make sense to scrape the lego website where you can select all their bricks.
Each named brick has 8 images (various orientations):
...This also makes me wonder whether it would make sense to set up multiple cameras, to get multiple orientations for every block. Use the image that has the lowest classification error, but use all orientations to train the network.
The images are not very useful to me because the images the machine takes are very different from those. But maybe it would somehow contribute. I'd be very ware of using other people's copyrighted content though.
Point taken re copyright.
Re similarity, it may be possible to write some sort of imagemagic filter that makes the bricks appear more realistic somehow. (But yeah, this would probably only be useful as a starting point for training the very rare bricks)
https://shop.lego.com/Pick-a-Brick
I find it hard to believe that there are only 1421 bricks, including color combinations.
Example: "Apple with leaf", Color Family: Dark Green, Exact Color: Bright green, Category: Foodstuff Element ID: 4107050, Design ID: 33051, orientation 8: https://sh-s7-live-s.legocdn.com/is/image/LEGOPCS/4107050_s8
...This also makes me wonder whether it would make sense to set up multiple cameras, to get multiple orientations for every block. Use the image that has the lowest classification error, but use all orientations to train the network.