A321 Flight crew reported loss of aircraft control during pushback procedure.

Narrative

After pushback; the ground crew stated; Set brakes." I set the brakes and responded; "Brakes set; cleared to disconnect interphone."Before disconnecting; the ground crew asked if we were too close to the road and whether I wanted to be positioned farther back. I confirmed and requested to move back an additional 10 feet. As soon as I released the brakes; the ground crew immediately instructed; "Set the brakes;" as we began rolling back. I quickly applied the brakes and set the parking brake after coming to a full stop. We rolled back only a few feet.Cause: After setting the brakes; I should have requested confirmation before releasing them; even though we were still in the pushback process. Treating brake setting and releasing like the transfer of aircraft control between pilots--using positive confirmation from both parties (e.g.; "Your aircraft" / "My aircraft")--would help prevent this issue in the future.My call to the ground crew should have been: "Yes; let's move back 10 more feet. Confirm cleared to release brakes." Only after receiving; "Cleared to release brakes;" should I have released them."