Air carrier flight crew reported a loss of aircraft control due to Inertial Reference System failure. The crew regained control and continued the flight.

Narrative

Just prior to reaching the waypoint ZZZZZ at 31000 ft; during a lavatory break; while I was reentering the flight deck; the autopilot disengaged and the aircraft started to descend and turn. This was caused by a IRU failure (as indicated by an IRU FAULT light and associated Cautions from systems dependent on heading and altitude from the IRU). We lost heading and attitude information on the CA (Captain)primary display and heading on the standby ADI. The CA took control of the aircraft from the minor upset and brought the aircraft back up to the cleared altitude. The aircraft descended approximately 550 ft below our cleared altitude during the event. We ran the associated QRH checklist for the IRU fault and recovered attitude and heading to the CA side using the IRS Transfer Switch. During the failure; the heading and wind information on the Nav displays appeared to be inaccurate; so we coordinated with ATC for confirmation of our heading and position relative to our flight plan course; and advised them of our equipment failure. ATC descended us below RVSM airspace and we continued to our destination. ATC advised that they did not have an indication of our altitude deviation. Over time the Nav displays and wind information eventually became reliable. We decided that it would be better that the FO land the aircraft due to the loss of flight director guidance on the CA side. The approach and landing was hand flown to ZZZ in accordance with the QRH without further incident. Cause: Equipment failure appears to be the cause. No recommendations to mitigate risk.