It does sound like you're overthinking. It's always a balance. There are projects where I've spent way too much time making sure it would be more flexible for the future but ended up never needing to add any of the functionality I had anticipated.

If it's working now with the refactoring you've already done, I'd say ship it as is and leave some comments in the code to remind yourself what your plans were. Then walk away from it for a little while, not necessarily 2 years but give it a week or so. Just let everything process on a "background thread" in your mind and eventually you'll figure something out and want to come back to it.