A319 Captain reported a loss of control and poor communications during repositioning at a gate with an inoperative jetway.

Narrative

The flight was scheduled on Aircraft Y; which had been written up by the inbound crew for several issues; including the write-up 'FO headset jack intermittent.' As we arrived at the aircraft; a mechanic arrived almost simultaneously. The CA and FO tested their headsets to determine if the issue affected both sides. The FO found that moving the headphone jack caused the audio quality to vary. Both the CA and FO heard loud audio feedback; preventing safe and effective communication. This feedback was also audible over the cockpit speaker. The maintenance technician advised he would be MELing the ICS (intercommunications system). I contacted dispatch to inform them of the impending MEL and; as such; declined the aircraft for safety reasons.We were assigned another aircraft and gate; to which we proceeded. Catering did not service the new aircraft; but the FA-A advised me the crew had enough catering to conduct the flight. They were an on-call crew and would take the aircraft back several hours later. After a short delay in ZZZ1 because the gate agent couldn't move the jet bridge away from the aircraft; the flight pushed back and took off. We arrived in ZZZ at approximately XA45 and stopped short of the gate; awaiting ground crew members to guide us in; as they were assembled in a huddle just below the jet bridge. After 2-3 minutes; the wing walkers and guide person were ready to receive us and lit their wands. The aircraft came to a complete stop; and flows and shutdown checklist were completed. I observed the jet bridge moving already; so I got up to open the door and use the lavatory. After exiting the lavatory; I found the jet bridge not connected and the door still closed; so I returned to the flight deck and saw the jet bridge operator repeatedly attempting to align jet bridge XX with the aircraft. After about 10 minutes of watching this; I made a quick announcement over the PA to let passengers know they were having difficulty with the jet bridge; then I opened the flight deck window to communicate with the crew on the jet bridge. I was told the last part of the jet bridge couldn't be rotated; so they were attempting to reposition it for safe deplaning. I asked; 'Do you have another gate?' 'No' was the response.A gentleman arrived minutes later. I presumed he worked for the airport authority; given the logos on his clothing. He repeatedly attempted to get the jet bridge to work; then stopped to diagnose the situation by looking in panels; shutting off power; and inspecting the exterior from the ground. I advised dispatch via ACARS that the door was not yet opened as the jet bridge was broken. We attempted to reach Operations via VHF; but no one responded. Dispatch also tried calling Operations by phone; but no one answered. I leaned out the window again and advised the ground crew member on the bridge to get a tug and tow us to a new gate. The passengers' frustration grew as the minutes passed. After being at the gate for over an hour; a tug pulled in front of our aircraft. I spoke with the FA-A and advised her of my concerns that we were approaching a critical threshold; with approximately 20 minutes before the time period. She then informed me that since catering did not service us; we probably didn't have snack supplies boarded. The FA crew would have enough snacks but likely not enough water for everyone at this point. I advised her the tug had arrived; so hopefully it wouldn't be necessary.After connecting the aircraft; the crew made preparations for movement. The FA-A asked how long we had for the clock; and I said we were getting close; but we had about 10 minutes; and they were about to start the push. All passengers were already seated at this time; with checks completed; and we were prepared for pushback. After releasing the parking brake; we commenced push-back. While being pushed; the tug operator asked; 'Would you like to be pulled to the new gate or go under your own power?' I replied we'd like to be pulled to the gate. After being pushed some distance from the block; the tug slowed to a stop. Multiple ground crew members converged under the nose of the aircraft; out of our view; and we heard a sound like they disconnected their headset. Several minutes later; one ground crew member used hand signals to indicate 'Chocks In.' Confused; I signaled him to put on the headset; but no one did. Unsure of their intentions; I guarded the brakes to prevent unanticipated aircraft movement. Both flight deck crew members used their flashlights to gain the ground crew's attention; hoping they would look up or communicate via the headset. We heard and felt the tow bar disconnect from the nose; and the aircraft rolled about 1 foot under its own weight before I arrested the movement and set the parking brake; which was neither requested nor communicated by the ground crew. We remained stationary at this point; which is when we passed the 90-minute DOT arrival threshold. We stayed in this location for what I estimated was 10 or more minutes before I got the ground crew's attention to put on a headset and ask what was happening. They then reversed the tug 180 degrees to reattach the tow bar and tug us to XY. Eventually; the tug operator came back on the headset and asked if we were ready. I asked her if she wanted me to release the brakes; and she replied yes. I released the parking brake; and we were towed to XY; blocking in at XC40. Dispatch was notified via ACARS of the new IN time. A 55-minute flight that evening turned into a 3-hour; 24-minute ordeal; with nearly 2 hours spent waiting for the ground crew to enable safe deplaning.After finally reaching our new gate; I stepped downstairs to the ramp to speak with a ground crew member to better understand what happened. The person I spoke to was the tug driver; who was the lead. I informed her that the lack of communication made this a challenge and posed a lot of safety concerns. The tug operator stated she tried to pull the aircraft backward but couldn't; so she had to turn it around. She added that she had never driven that tug backward. Before we got off the plane; a flight attendant from the aft cabin informed me that a passenger had called the airport authority directly while still at the first gate (XX); and allegedly; the person on the phone placed the blame squarely on our airline; whereas their Jet Bridge was not operating properly. Multiple passengers overheard this exchange; and their reaction was negative; as they were already very frustrated by the situation. Throughout the course of this ordeal; I made several PA announcements to keep passengers informed and updated as they grew increasingly restless and frustrated. Needless to say; they were less than impressed; and that's valid; we let them down and under performed.After pushing back from the gate; it appears the left side wing walker used a tug possibly instead of walking by foot. When we came to a complete stop before they disconnected us I seen him exit. Tug operator not remaining in constant communication with pilots. Hand signal 'Chocks in' was not the appropriate sign and was given by hand with no illuminated wands. Little information passed directly to flight deck when they knew they had issues with the jet bridge; they need better communication.