An ERJ-170 pushback crew failed to positively communicate visual signals with the flight crew during pushback after their headset failed.

Narrative

After closing the cabin door we were unable to establish communication with the pushback tug operator. After several attempts one of the ground personnel walked into our view and motioned that their headset was not working. I proceeded to give him the hand signals for brakes set; pull chocks. The operator acknowledged the signals and proceeded back to the tug. Very shortly after the tug operator was out of sight the First Officer and I felt bumps from below; that to me felt and sounded like the tow bar being moved. I released the parking brake but kept pressure on the toe brakes; as it became more obvious that they were going to push the aircraft I released pressure on the toe brakes as well. Once the push was complete there was no query from the ground crew as to the status of the brakes; and the only other hand signal received was unknown (both fists interlocked above the head). The ramp was uncontrolled and there were no other aircraft behind us. The push back was completed without incident; but not according to SOP... No one was injured and no damage occurred; however the potential for both were there. Ground personnel should be familiar with the pushback procedures and the hand signals set forth in the FOM. They should also be instructed not to initiate a pushback when the red pushback indicator light on the nose landing gear is illuminated without prior consent from the Captain.