Air Carrier Captain reported a miscommunication with ground crew during push back led to the ground crew disconnecting the tug prior to the pilot setting the brakes causing the plane to roll about two feet.

Narrative

We pushed back from Gate XX; when we stopped momentarily before being pulled forward he said 'clear to start' (we started the engines while being pulled forward) then he pulled us forward to abeam Gate XX and we stopped. I had my right hand on the parking brake and my left hand on the push to talk on the side stick like I always do anticipating the ground crew to tell me to set the brakes. The next thing I felt was the tow bar being disconnected and I looked down to the signal man and he was signaling to me to set the brakes and that's when I felt the aircraft starting to roll. I set the brakes and we stopped after rolling approximately 2 ft. At that point I flashed the nose light a few times to get their attention because I wanted them back on inter phone. The signal man and gear pin man we're trying to get a salute out of me and I kept requesting that they plug back in with my hands on my ears. They finally plugged back in and I asked them if everyone was ok. I said; you never told me to set the brakes and He said yes I did and you (meaning me) told him to disconnect the inter phone which was a complete lie because the First Officer and I looked at each other and said to each other that I never said a word. I would never set the brakes until they told me because sometimes they pull you forward. And then while I was taxiing out for take off; I got a message from Dispatch not to take off and I need to call the Duty Chief. I called Dispatch and he patched me to the duty chief and he asked me if I was feeling OK. Suggestion - Create a double read back with ground crew