Hash the phone numbers, and compare hashes to check for abuse. This way you don't know the phone numbers, but you can ensure nobody reuses the same one twice.

Thanks for making this open-source, and implementing the client-side key generation as I recommended on GitHub.