Most games don't do any heap allocations at al during runtime(or at least very very few), so you shouldn't/don't end up with garbage collector pauses during normal gameplay. If you refcount your main structures, they can be auto cleaned up /removed when changing levels/loading scenes/swapping chunks out on the fly.