My wife worries about cutting things back too. My father-in-law told her not to worry. I cut things back but timing is important. I usually prune early in the year while the plant is dormant yet late enough so new growth is not damaged by cold temperature. I cut Crepe Myrtle way back and it grew back bigger and better.