Captain reported releasing parking brake while waiting for ground personnel to locate an operable headset and prior to tug being hooked up to aircraft to minimize on-time departure delay. Aircraft rolled back 4-5 feet before Captain noticed ground personnel waving at which point Captain reset brakes.

Narrative

Shortly after the Ops Agent confirmed we received our takeoff performance data right at push-time I observed the Ground Crew getting into position after removing the ground air and power. I could see the Ground Crew attempting to establish contact and I attempted to reach them several times. It appears the first headset they tried using was inoperative and 1 Ground Crew Member went off to find a different headset. Before actually establishing voice contact with the Ground Crew; I released the parking brake to help get close to an on-time push-back (3 minutes late at time) and the aircraft began rolling backwards. I did not immediately notice the aircraft moving due to the jetbridge also moving away from the aircraft at the same time (optical illusion). Another Ground Crew Member waved his arms vigorously and advised me that the aircraft was moving so I applied the brakes immediately. Apparently; the Ground Crew had not yet connected the tug and towbar but did remove the chocks early without advising the Captain. Once on intercom the Ground Crew advised me that the aircraft rolled from the -800 marking to the -700 marking which I believe is approximately 4-5 ft. No injuries were reported on the ground or in the aircraft. The Ramp Supervisor also came on intercom after a few minutes to ask what had happened. We discussed the sequence of events described above and she advised me to submit a report. Once the tug was connected the rest of the push-back and the flight to ZZZ went normally.I need to ensure proper communications with the Ground Crew before releasing the parking brake in all cases. I need to not let perceived time pressures affect my decision making and keep the brakes set until both Ground & Flight Crew are certain we are ready for push-back.