Background
Someone on the Internet claimed to have achieved 1M+ concurrent connections on a single EC2 instance. While I can not verify this claim, it is true that EC2 does not set a hard limit on the number of connections it can handle, except in one case, that is throttling of security group connection tracking.
Security Group Connection Tracking Throttling
Buried in the EC2 User Guide is the throttling behavior of security groups, which are essentially stateful firewalls. If you have a lot of connections on each EC2 and your usage pattern satisfies one of the following conditions, you may be hitting the limit the hard way (packet drop).
- If specific IP ranges (i.e. not 0.0.0.0/0 nor ::/0) are set in a TCP or UDP security group rule.
- Connections go through NAT gateways, Network Load Balancers, PrivateLink, etc. Complete list is in docs.
- Both inbound or outbound rules on the security group have set IP or port limit for the relevant protocol. This is less likely to happen because few would modify the default ALLOW ALL outbound rules.
Unfortunately, the maximum number of connections that can be tracked per instance is a secret not provided in AWS documentation. We can only get conntrack_allowance_exceeded numbers from the ENA driver, which is non-zero if packets are dropped because of this.
Conclusion
If you maintain a large number of concurrent connections per EC2 instance, avoid connection tracking on security groups if possible, and keep an eye on ENA driver metrics to spot problems early.
Thanks for reading. See you next time!