I think of my butterfly garden as a place for bees, butterflies and hummingbirds. But wasps attack caterpillars, especially paperwasps, yellowjackets and hornets. So you will want to place the host plants where caterpillars are likely to hatch and feed away from the area bees and wasps will be feeding.
While bees do not attack caterpillars like their cousins the wasps, the air movement of bees wings nearby triggers sensor hairs on the caterpillars. The caterpillars then sense a predator nearby and freeze, or drop off the plant.
On the opposite side, if you wish to keep caterpillars away from your peppers and other vegetables, plant them close to the flowering plants in your yard that attract bees. My Mexican heather is covered with bees about 9 months out of the year. Consider a border of heather or some other plant loved by bees about your vegetable garden to discourage caterpillars from hanging out.
It has recently come to my attention that Imidacloprid is extremely toxic to honey bees. If you must use pesticides in your garden be sure to read up on this one first.