Differing inputs causing verification failures is indeed an issue. In addition to data access races, replication latency also causes this. The legacy service always reads from the primary MySQL instances per shard, but the new service always reads from replicas for scalability and geo distribution.
One slightly helpful mitigation we have in place relies on a data versioning system meant for cache invalidation. The version is incremented after data changes (with debouncing). To reduce false negatives, we throw out verification requests where the two systems saw different data versions. It's far from perfect, but it's been effective enough.
One slightly helpful mitigation we have in place relies on a data versioning system meant for cache invalidation. The version is incremented after data changes (with debouncing). To reduce false negatives, we throw out verification requests where the two systems saw different data versions. It's far from perfect, but it's been effective enough.