Air carrier Captain reported discovering after pushback that dry ice was on board without proper notification. The flight crew returned to the gate and the dry ice was removed from the aircraft.

Narrative

During pushback; we receive the load closeout revision XX XA:23. The FO receives the closeout and begins reviewing as I focus on the pushback operation. The FO then asks if we have a NOTOC for restricted articles onboard. I replied we did not receive any notifications for restricted articles. I ask the Pushback Driver and he said he 'didn't think so.' The FO calls Operations who said there were no restricted articles loaded. About that time; the Tug Driver calls back and said he thinks the Crew Chief loaded dry ice in the aft cargo. I said I need the NOTOC paperwork or the dry ice removed before we can proceed. We sent a message to Dispatch who replied; 'Just talked to Operations and they verify there is no NOTOC on board.' The Crew Chief later confirmed he loaded 2 boxes of dry ice. I tell the FO to work with Station Operations to have the dry ice removed. We eventually agree to be towed back to the gate and the dry ice was removed. This voided our closeout and we were delayed awaiting revision XY arriving at XB:10. The threat is not notifying the flight crew of the presence of hazardous materials on board.Dry ice was loaded in the aft cargo without informing the Captain; or Dispatch; and providing a NOTOC. Only the FO; following proper procedures; reviewing the load closeout caught the error and saved us from unknowingly transporting undocumented HAZMAT. Stress the critical importance of loading HAZMAT and informing the crew.