I haven't had a problem as there are rubber pads against the post. The clamp holds evenly around the circumference and and the 4 other attachment points take most of the torque off the clamp and post. There are racks that are just the seat post clamp which I think would be where damage could happen easier.

The panniers are great and can hold a lot!