Enroute Controller described an unsafe condition noting the ATOP equipment failed to note needed estimates when traffic was in holding; adding that failed pilot reports contributed to the confusion.

Narrative

I assumed the sector and was briefed that Aircraft X was approved via VOR A..VOR B..VOR A with a request to hold at VOR A for about an hour. I contacted the previous sector to determine the aircraft's clearance limit. I was informed it was VOR B. I assumed control of the data tag and requested that the pilot inform me entering and exiting the hold. I established holding airspace via ATOP. Aircraft X informed me he was entering the hold and would be exiting at XA35Z. I took a break at about XA20; and briefed the relieving Controller of the situation. At about XA55z I returned to duty and was assigned back to the same sector. The Controller advised that the aircraft had not reported leaving the hold; and had extended the reservation. After assuming the position; I examined the aircraft's messages and saw that he had reported over the hold fix and estimated the next fix on his flight plan. This fix was the coordination fix with the previous sector and his estimate was in about 5 minutes. I quickly coordinated with the previous sector; and updated the processing to reflect to position of the aircraft. The aircraft was untracked by ATC on his return from his hold; and was close to violating the previous sector's airspace. Recommendation; I believe that ATOP did not show the message of Aircraft X estimating the next fix to the Controller working the sector. ATOP assumed the aircraft was making an operational normal report and ignored the next fix estimate because the aircraft was in the hold. If this was the case; it is a dangerous limitation of ATOP and should be addressed. I might have led the aircraft down the path when I asked him to report leaving the hold; but he shouldn't have left the hold without an ATC clearance; as the hold fix was his clearance limit.