A319 Captain reported aircraft rolling back after being told by ground crew to release brakes. Other mental lapses by this ground crewman were also noted.

Narrative

Cleared for pushback from gate; referencing inbound. Cleared to release brakes by pushback crew. Aircraft started to roll back; tug driver said 'Set your brakes;' you're starting to roll away.' I stopped the plane using the brake pedals; so as not to launch flight attendants during their demo; then set the parking brake once the aircraft came to a stop. Asked pushback crew what happened; was told 'I don't know.' Tug came back up to plane; and we clearly heard them attaching the tow bar. Completed pushback; was told 'Towbar disconnected; pin removed;' but ECAM still showed NWS DISC. Told pushback crew; who then removed the NWS pin. I am convinced the towbar wasn't connected to the plane at all the first time. I think there would have been some noise from it popping loose. For what it's worth; this same Ramper was the second guideman for the dogleg turn-in to park; and didn't seem to understand that there is a bit of lag in straightening out the nosewheel coming off a full-deflection turn. This is a tricky gate...maybe a bit more training might be in order.