Air carrier Captain reported advising the tug operator brakes released for the push back but the brakes were set resulting in a broken tow bar and aircraft damage.

Narrative

At gate XX; after aircraft ready for push FO calls for push; tug operator gave his intercom speech and asks for brake release. I released bake and we are told to hold the push for an aircraft behind us. I tell the tug operator to hold the push for traffic. 3 to 4 minutes later ramp clears us for the push and I tell the tug operator brakes off clear for push as I instinctively flip the brake handle inadvertently setting the brake. As the push began I felt resistance and looked at the brake indicator and realized the brake was on; I immediately turned break off and aircraft rolled a few feet back. I told tug hold push. Tug operator asked me if the brake was set; I said it is off now but it was. He said you told me brakes released ; cleared to push. I asked him if there was any damage and he stated the tow bar was broke. I told him I'd call maintenance. Maintenance came and confirmed the tow bar was bent and the pin on the nose wheel it attaches to was slightly bent so the aircraft would be taken out of service.Cause: Because we were still at the gate in original position I reverted to my muscle memory for brake release the second time and did not check the indicator. Tug driver apparently did not check for nose wheel brake light off and took my word for it.