Yes, maybe its caterpillars. The caterpillars of some butterflies can be quite a nuisance, while the butterfly itself does no harm.