A319 flight crew reported brakes were not properly set upon arrival at the gate; resulting in momentary aircraft loss of control.

Narrative

Upon arriving at Gate XX; the captain had stopped the aircraft about 0.5 sec after the STOP was annunciated on the self-park gate screen. It appeared to me that he was aligned and stopped when instructed. As the aircraft stopped; I saw the captain move the parking brake handle while I shut down the left engine; ran my flow; and retrieved the parking checklist. Although I saw the captain move the parking brake handle; I did not verify it was positioned to 'Set'; nor did I see the brake pressure before initiating my shutdown flow. After I started the checklist and announced 'parking brake...' The captain replied; 'Set. Pressure normal.' However I noticed the pressure was not normal and said; 'No it's not;' while reaching and pressing the toe brakes. That's when I noticed the aircraft was moving aft. Pressing the toe brakes stopped this movement. The captain also pressed his brakes as well. We both looked down and saw the parking brake was not set. He mentioned that he 'definitely had set it.' While he said this; I went to return to the checklist but he told me to stop as he wanted to see what was going on outside and kept looking at the jet bridge agent who was shaking her head as if the position of the aircraft was insufficient for the jet bridge. This caused the captain to try to communicate with her including opening his window. I insisted that we continue with the parking checklist before we figure out what's going on outside as both engines had been shutdown. He said no; he wanted to determine if they need to move us first. I proceeded to do the checklist myself; but he then interrupted me saying we have to do it together. I again insisted that we've already had one error; let's complete the checklist first to ensure we don't have another error; then we can figure out what's going on outside. He agreed and we completed the checklist. Ground personnel were clear at the time of unintended movement and no injuries (that we know of) occurred. However; the aircraft moved enough that it had to be towed back into position. The towing occurred uneventfully.