Air carrier pilot reported after push back the tug driver stated release brakes resulting in the aircraft rolling because the tug had been disconnected.

Narrative

We were getting pushed out of ZZZ gate XX. We were cleared to push to spot XX in the 1 alley. The initial push followed SOP. In the alley facing south; the ground crew in the tug said brakes set." I responded with "Brakes set; cleared to disconnect interphone." The ground crew in the tug then came back on and said; "release brakes." I then responded with "brakes released" and released the parking brake. The left guide man still had his wands crossed. I asked that ramper in the tug again "verify brakes released?" He responded with "release brakes." I responded with "brakes OFF" to clarify the current status.Then the ramp crew disconnected the headset and the airplane started to move as the tug was pulling away. When it was clear the tug wasn't pulling the airplane I applied the brakes and signaled the brakes were set using hand signals to the guide man to my left. The ramper then showed me the gear pins and left.This is following 3 days of issues with ground crews in ZZZ. The ground crew would push onto Taxiway 2 from the 1 alleyway but would not push the airplane far enough back to allow airplanes to enter the ramp via the 1 Taxiway. This personally happened twice to us; once where we had to release our brakes and get pushed back further because the ground crews did not push us to the right spot the first time. So the potential that brakes would have to be released after being set was not unexpected in ZZZ since there had been multiple issues with pushes and the location of where the crews were dropping aircraft. We know this is an issue because ZZZ Ground was telling ZZZ airport operations to talk to the push crews because of repeatedly not pushing aircraft to the right location."