Air carrier Captain reported being notified during taxi of a Lithium battery powered electronic device which was improperly configured for transport in the cargo compartment. Flight was delayed to retrieve the electronic device and flight departed safely.

Narrative

After brake release a Flight Attendant called the cockpit and informed us that the passenger in 20D informed them that a GPS tracking device in his checked luggage indicated that his bag was in a different location than the aircraft. He was concerned and wanted to verify that it was indeed in the belly of our plane! We checked with ZZZ station operations who verified that the bag was indeed scanned as part of cargo at the time it was loaded in the belly. At about the same time it was dawning on all of us; what is powering this device? (I had not personally heard of such a device; nor did I know of its size or composition). The Flight Attendants inquired with the passenger who looked it up and indicated that the device was powered by a lithium battery. As this goes counter to our safety policy regarding lithium batteries with its presence in cargo and having not been vetted through the HAZMAT process or TSA screening as a carry on; we were posed with a dilemma. After some deliberation on our part and with the passenger's willingness to provide the code to unlock the lock; the ramp personnel retrieved the GPS tracker with its lithium battery and handed it through the cockpit window. The device was retrieved in the presence of a ramp supervisor and a station agent. The First Officer and I were able to visually verify that all the passenger's items that were taken out of the bag were again placed back in; save for the tracker. We verified with the ramp personnel that the lock was also again securing the bag. The First Officer handed the tracker back to the Flight Attendants who gave it to the passenger. The device was a small metal object about the size of a half dollar with an Apple logo on it. No additional comments.I don't think that adequate procedures are in place to prevent this and too much trust is placed in the flying public. I also think that additional guidance is needed in the FOM to address this specific issue. I had no way of knowing if the device was in compliance.