You mentioned "it's sits a little higher than the shaft", as if you weren't sure that was a good thing. On the contrary, that is exactly what it is supposed to be like. That's how it "keys" into that groove in the impeller. I'm still not sure if you removed the impeller or not to "tap" the key into position. I can't see how you can do it any other way, though. But I'm coming back to, again, that most likely the impeller was installed in the wrong direction the first time. As I mentioned above, if the key WASN'T making contact with the impeller (if it was vertically higher than the impeller)... for one, you definitely would have noticed (or should have). And, two, as soon as you started the engine up, the key would have flung away.
I HIGHLY doubt the slot that the key fits into is worn like you're thinking. Like above, you would have noticed something like that. Besides, the IMPELLER "center" would have to be the one that was extremely worn for the key to slide out. But even that, that's pretty much something that will never happen.