A320 Captain reported the aircraft rolled after pushback and before the ground crew had cleared the area; even though the brakes were set.

Narrative

After pushback and engine 1 start; tug driver informed me that push was complete and asked me to set the parking brake. I set the brake; visually confirmed that the pressure increased on the brake pressure gauge; and the green ecam memo park brake" appeared on the upper ecam display. Shortly after the first officer said "engine one stable" the tug driver said "Whoa; what's going on up there; you guys are rolling!" Simultaneously we felt and heard a clunk from under the nose. I quickly stepped on the toe brakes but at this point we weren't moving anymore. I asked if everyone was ok and the tug driver responded that everyone was. I kept my feet on the toe brakes and kept them depressed; while trying to assess the situation. I noted to the first officer that the park brake handle was set; the park brake memo was still displayed; but now the brake pressure gauge and accumulator were now reading zero. I recycled the park brake handle; and after doing so the accumulator pressure immediately jumped into the green range and the brake pressure now read 2000psi for both the left and right wheel brakes. I tried to communicate with the tug driver again and now all I could hear were clicks over the interphone. Shortly after he appeared and gave the signal that his headset was no longer working. After some discussion with the first officer and with our park brake being unreliable at this point; I concluded that the safest thing to do was have the ground crew clear the tug and tow bar out of the area and pull back into the gate under our own power. I coordinated with our dispatcher and the fo coordinated our gate return with ground ops. I pulled back into our gate very slowly and kept a close eye on our brake pressure. After returning to the gate I wrote up the park brake failure and nose wheel striking the tow bar. After getting home and having the benefit of hindsight; I feel like I probably should have shut down the engines and had the ground crew reconnect the tug and tow bar and pull us back into the gate. I could tell that the tug driver was rattled by this incident of our aircraft rolling into the tow bar; and now the situation was complicated by the fact that his headset was no longer working. I just wanted them out of the way; but I realize now that shutting down the engines and being tugged back to the gate would have been the more conservative approach."