If we all came from Adam and Eve....?

...then why didn't we die out after say...4 generations of inbreeding, or better still actually evolve backwards into plants? Surely there's a flaw there? Did I miss something?