An A320 Captain reported that an unauthorized jumpseater was on board but the Gate Agent refused his command to return the jetway thus ignoring Captain's authority.

Narrative

Captains Authority. After a very hectic last minute boarding with transferred passengers from a mechanical delay on another flight; a Company pilot arrived as door was getting closed. The cabin was full and so I asked for his Flight Deck Authority Access Card. He did not have one. So I tried to quickly get the CSR's attention as he was pulling the jetway away to come back. The concern was that the CSR did not comply with my clear hand signal to come back and instead only stopped the jetway and made a phone call. This is by no means a critique towards the CSR but a critique on the misguided policy of Company gate departure procedures that place a heavy burden on CSR's to get the flights out on time and not get charged a delay. The serious concern here is that my Captain's authority was completely ignored and deviated from; when the need for the jetway to return was requested; as this could have been for a myriad of possible emergencies to include; unathorized/unkown person on board; medical emergency; terror threat; fire; smoke; fuel threat; etc. The responsibility lies with the Captain and thus his or hers decision to return the jetway should not be questioned when it is clear they are signaling for it to return and thus the CSR should not be in any way charged with any delay whatsoever. The delayed action to return the jetway presents a liable and dangerous scenario!