A TRACON Controller reported they vectored an aircraft off the final approach course to avoid unidentified VFR traffic but the aircraft turned the wrong direction resulting in a NMAC.

Narrative

Aircraft X was cleared for the RNAV Straight in Runway XX at ZZZ at 026 ft. Aircraft Y was maneuvering west of the final approach fix (ZZZZZ) VFR squawking 1200 unverified and not on frequency. I then saw Aircraft Y heading towards ZZZZZ and issued traffic to Aircraft X. Aircraft X responded looking for traffic. I called ZZZ Tower to see if they were talking to the aircraft and they said they were not. I reached out to the VFR aircraft to see if they were on my frequency and they were not. I got the callsign of the VFR aircraft from the ADSB readout. Once I noticed Aircraft Y was still heading east towards the FAF at 2500 ft. I then cancelled Aircraft X approach clearance instructed them to maintain 2600 ft. and to turn right heading 090. Aircraft X then readback the instructions verbatim. I had other work to do and inbounded other aircraft to ZZZ Tower per the LOA. While I was on the shout line with ZZZ Tower I heard Aircraft X say traffic in sight. When I look up again at the scope I see Aircraft X turning WEST with the CA/CA (Conflict Alert) alarm going off. I verified with Aircraft X that if he was turning to the east. He said yeah; we're turning west. Targets merged with 100 ft.indicated separation. I then told hold him when he's done with his maneuvers to fly heading 270. He then said that he was flying 090 then mid sentence realized his error then said sorry. He later seemed shaken. While he was flying west he then began to descend below the MVA. I had to correct him for that too. I had my Controller in Charge reach out to ZZZ Tower and I guess they got a hold of Aircraft Y and then he checked in with me for flight following to ZZZ. He said that he tried to enter ZZZ Tower airspace and they told him to remain out. Something doesn't add up. Why would ZZZ Tower say they didn't talk to Aircraft Y when the pilot says he tried too. ZZZ has a large flight school and they all like to loiter around the airport VFR. Especially near the final approach courses and FAF. ZZZ Tower is a new facility. They do not have an ASDE or access to a FDIO (Flight Data Input/Output) [system]. Our workload for ZZZ inbounds and outbounds have tripled. Adding the fact that we now have to be hyper vigilant for VFR traffic loitering within such close proximity to the airport is unsafe. My suggestion is that aircraft not loiter near active final approach courses or such close proximity to the airport. The VFR aircraft need a practice area for maneuvers away from the airport; away from arrivals and departures.