C172 pilot reported they had unexpectedly little clearance flying over a tower during climb out due to high density altitude and passenger load. Pilot deviated laterally and continued the flight.

Narrative

I was flying public benefit flights with passengers. I completed W&B and performance calculations before the flight and all checked out within limits. On the takeoff roll for the first flight I noticed a much longer ground roll than I am used to and the climb was super slow at Vx (as little as 150FPM). This put me within about 80 feet vertical of an obstacle/tower (675AGL) we were using to navigate our route visually; and much lower than I would normally be while exiting the delta airspace. I noticed we were lower than I was comfortable with so I deviated to the right and had approximately a half mile clearance from the obstacle. The rest of the flight was uneventful.On the next flight I requested a lower passenger load that would put me even more below MTOW and again within limits. I had a better ground roll; takeoff; and climb rates of approx 300-350 FPM; which I still was uncomfortable with. The rest of this flight was uneventful; but due to the high density altitude I decided to call it a day and not perform further flight. I learned that even though the performance numbers and W&B check out; you still may not get the performance you expect or need during high density altitude days. I also suspect that the organization running the flights relies on self reporting of passenger weight; and I may have had a higher load than I thought I did. In the future I will have a much lower margin below MTOW to account for this and/or use my own scale to weigh passengers.