Air taxi Instructor with student reported a near miss in the traffic pattern at non-towered airport.

Narrative

During a training flight; myself and my student were flying back into ZZZ Airport. We had just canceled our IFR flight plan; as we obtained visual contact with the airport and were going to enter the left downwind for runway XX. When we switched to the advisory frequency for ZZZ; the only aircraft that was communicating on frequency was a helicopter; that was turning base to final while we were entering the left downwind for runway XX. There was another target that we observed on our internal TAS system that appeared to be in the pattern as well; but we did not head on the advisory frequency. As we were much faster than this aircraft; we gained on it very quickly; and decided to separate from it's space by circling in the downwind to give the aircraft time to re-gain separation from us.This event was discovered because our FDSO was alerted to a close encounter from our aircraft with another. In the future; in order to prevent situations like this from occurring; we should be more proactive when discovering that we are unable to properly communicate with aircraft in the area for an un-towered airport. We were in communication with the helicopter that had plenty of separation to land; but we were unable to communicate with the other aircraft in the pattern. As soon as we realized this; we should have advised the CTAF frequency that we were switching back to the approach frequency to request aid to separate from this traffic. At the least; this would have made them aware of the issue; and likely allowed us to receive delayed vectors on a new squawk code to allow the spoken for traffic to pass. I believe that earlier recognition of a communication problem is necessary; and would make sure that an event like this would not happen again.