Emb170 Captain experiences failed communication with the ground crew during pushback; resulting in the parking brake not being set when the tow bar is disconnected.

Narrative

We were scheduled to push at XA:12 but had no push crew. Calls were made to Operations notifying them we had no push crew and the aft cargo door remained open. At approximately XA:23 a push crew arrived and the tug driver made visual contact with me through my DV window by tapping his headset indicating that he could not talk to us. Visually indicating by tapping my own headset and shaking my head I communicated to him that we would not push with an inoperative headset. He gave me the signal to release the brakes which I refused to do given that we did not have 2-way communication and the aft cargo door remained open. He either switched headsets or wiggled the connection and restored 2-way communication. I verbally told him that I could not release the brakes until the aft cargo door was closed. He told me he thought it was already closed. I told him it was indicating open and he needed to visually inspect it. He left and the aft cargo door was then closed and indicated such. The tug driver verbally told me everything was set and we were ready to push. We pushed straight back and were stopped. The marshaller with the wands crossed the wands to indicate we were not to move. I had the cockpit dome light on so we should have been clearly visible to the push crew. No verbal or visible command had been given for us to set the brakes. I verbally queried if we were to set the brakes and received no response. I gave a second inquiry and go not response. I presumed verbal communication had been terminated at that point. I made eye contact with the marshaller with the crossed-wands and visually inquired if we should set the brakes and got no response. At that point another push crew member came into view via my DV window moving toward the forward view window and he held the strap up in the air indicating it had been removed from the nose gear. I visually inquired of him if we were to set the brakes and got no response. At that point the tug reversed out from under the airplane proceeding backward toward the terminal and the airplane began to roll slightly forward (not more than several feet). I depressed the brakes and set the parking brake. The First Officer and myself discussion the safety ramifications of a push at night; with little visual reference in the dark that you might not even notice the aircraft was moving. I put my hands up indicating confusion to the ground crew about why we were never instructed to set the brakes at the conclusion of the push. he push crew reunited near the top of the 'T' and huddled to talk; perhaps conferencing about the incident. Prior to taxi from the ramp I contacted operations and informed them of the details of the incident and asked them to explain the push crew the safety implications of the situation. I informed operations I would passing the incident along to our Flight Ops management for follow up as well. The headset issue remains a problem in ZZZ and this incident if not for vigilant situational monitoring could have gotten ground personnel hurt or property damaged.