A319 Captain reported air condition pack was MEL'd which led to high cabin temperature due to a delayed departure. This resulted in a return to the gate and the aircraft was refused.

Narrative

Assigned a 319 with #1 PACK on MEL; I elected to accept the aircraft because it was cool at the gate and ground delays were not anticipated. I knew the aircraft would cool enroute and temperatures were more manageable on the ground in ZZZ1. We pushed back from the gate and started both engines. We then waited about 10 minutes for other traffic in the ramp area to clear.At this time it became evident that with the rising afternoon temperatures; the cabin temp in the front of the aircraft cabin could not be maintained. It was climbing; but fairly slowly. As we taxied; we received a wheels up time for ZZZ1; incurring a 20 minutes delay. We stayed on Taxiway 1 and I watched as the temp continued to climb; now in the 80s in the forward cabin. I called the flight attendant (FA) and they stated it was not comfortable; but it was manageable. We then taxied to the runway for our wheels up time. We were #1 for departure when Tower informed us of a ground stop to ZZZ1 due to weather. We taxied clear of traffic and at this time the forward cabin was showing 100 degrees. I knew we were too close to min takeoff fuel and would need to return to the gate for gas.I also knew at this time that until the evening temperatures came down; I could not take this aircraft again due to the inability to maintain temperature. I called the FA at they again stated it was warm in the front. At this time the FA said something needed to be done as the forward cabin was becoming uncomfortable. Upon return to the gate; we immediately deplaned to ensure passengers remained safe and in more comfortable conditions. I wrote up that the aircraft cannot maintain adequate cabin temps with Pack MEL. Maintenance then came back and stated that the only way to not have the aircraft assigned to this flight is to refuse it. At this point I wrote up the following: Crew refusing aircraft due to high cabin temperature / pack MELWe swapped into a different aircraft and this one was wisely taken out of service to fix the PACK.Cause: PACK MEL in summer conditions is not ideal. Although it is impossible to predict long taxi times; as a policy I will no longer accept single pack aircraft in summer months as a Captain.