A TRACON Controller reported an aircraft flew off course and below the Minimum Vectoring Altitude. The Controller delayed their decision to advise the aircraft due to the perception it may be due to false tracking of the target which is a recurring issue with their radar display software.

Narrative

Aircraft X departed ZZZ1 on the ZZZZZ departure. When the aircraft tagged up at around 500 ft.; he appeared to be turning northbound; away from the assigned departure procedure; directly towards downtown and the approximate 1100 ft. tallest building. Initially I was unsure if this was actually happening because sometimes STARS will show low altitude departures moving in directions they are not moving. The aircraft however continued northbound and appeared to level at 500 ft. The pilot finally made contact with me and I asked him if he was on the departure procedure. When it became clear he was not; I began advising him of the obstacles directly in front of him and that he had turned away from the departure 2100 ft. and not climbing fast so I could not vector him. I advised him of his low altitude and told him to climb and maintain 3000 ft. Once he reached the MVA I began issuing vectors on course. I gave the pilot a 'brasher' warning and the TRACON phone number.It would have been helpful to know earlier that the pilot was indeed turning off the departure. I have been conditioned to wait and see because of how often we get erroneous depictions of aircraft turning on departure or the target just bouncing around. We are not allowed to turn any north flow ZZZ1 departures until leaving 1100 ft.; so I think if ZZZ1 Tower Controllers were able to watch and notice if the aircraft was turning earlier than that; they could have informed me. I could have at that point tried to make contact with the pilot since he had not checked in yet. But beyond that I don't know what else I could have done differently. I didn't use emergency vectors because the pilot had reported the prominent obstacles in sight and was turning to avoid.