Someone on the Internet claimed to have achieved 1M+ concurrent connections on a single EC2 instance. While I can not verify this claim, it is true that EC2 does not set a hard limit on the number of connections it can handle, except in one case, that is throttling of security group connection tracking.
Buried in the EC2 User Guide is the throttling behavior of security groups, which are essentially stateful firewalls. If you have a lot of connections on each EC2 and your usage pattern satisfies one of the following conditions, you may be hitting the limit the hard way (packet drop).
Unfortunately, the maximum number of connections that can be tracked per instance is a secret not provided in AWS documentation. We can only get conntrack_allowance_exceeded numbers from the ENA driver, which is non-zero if packets are dropped because of this.
If you maintain a large number of concurrent connections per EC2 instance, avoid connection tracking on security groups if possible, and keep an eye on ENA driver metrics to spot problems early.
Thanks for reading. See you next time!