They aren't really variants. The idea of variants was that there was one timeline and anyone who deviated from that timeline was a variant.
In comics, its just a multiverse. They aren't variants, they're just different versions of characters. Variants really only apply to the Loki show.
But yes every universe has its own celestials. Usually they're consistent.