Air Carrier flight crew reported non-standard push back procedures used by ground crew.

Narrative

I was Captain on a flight from ZZZ to ZZZ1. We were at our gate. We were delayed about 6 hours due to an airplane swap because of a maintenance issue on our previous aircraft. We were given a hotel room near the airport for the extra time until our flight. We arrived at the aircraft about 40 minutes before pushback. The airplane was being boarded when we arrived. All preflight duties were normal. After loading was complete; the pushback was delayed while we were waiting for some weight and balance issues to be resolved. This was transparent to us. The tug driver just told me there would be a delay. The First Officer told me later that there may have been training in progress during the push as; during his walk around; he heard a man in the tug say that he needed a Supervisor to be with him. The beginning of the push was normal. Pushback clearance is not required on the ramp at ZZZ. The first thing I noticed was when the airplane was stopped during the push for much longer than normal without any communication. The tug driver did not say; 'Set brakes' therefore I did not respond. After a short time; both the First Officer and I noticed the aircraft moving backwards; so I hit the brakes and set the parking brake. I estimate we rolled back 10 to 15 ft. before I stopped the airplane. There was no 'breakaway' call. The tug driver was still on headset. He claimed that he told me to set the brakes. He did not - and I did not respond. He also never told me 'airplane disconnected; bypass pin removed'. After a long period of confusion; I did find out the bypass pin was removed. Because I never heard this; I did not clear him to disconnect headset. I did receive an all clear signal (lowering of the arm) from the Marshaller and I did flash the taxi light. The tug driver was backing away while this occurred. He then stopped and would not move the tug. I was working with Operations on the radio to try to get him to move out of the way as I could not taxi because he was still in my way. Operations kept telling me they were sending out a Supervisor to see what was going on. Finally the tug driver backed away and was clear of the aircraft so I could taxi. When the First Officer called ground for taxi; the controller commented how deep they pushed us back. Taxi out and the remainder of the flight was normal.