A B737-300 First Officer climbed through cleared altitude when the autopilot failed to capture and level off.

Narrative

It was the First Officer's leg; so I was flying runway heading to 5;000 ft. Before reaching 5;000 ft; we were given a turn to 210 and climb to 8;000 ft. During the turn; as we passed through 3;000 ft; I engaged the autopilot in Heading Select and Level Change pitch mode. During the climb through 7;200 ft; the approaching altitude horn sounded and was noted. At the same time ATC called out traffic that would pass 1;000 ft above us in a little bit. The Captain began searching for them. Before reaching 8;000 ft; the Stab Out of Trim light illuminated momentarily and I directed attention to it for a few seconds. As I returned my attention back to monitoring our progress I noticed the autopilot had not captured 8;000 ft; in fact it had reached 8;700 ft and was still climbing with no warning. Level Change was no longer engaged as well. I pulled the throttles back to idle and pushed the nose over as we reached 8;800 ft. At 8;600 ft; ATC queried us about our altitude and we told him we were correcting. We advised them that we had had a problem with our system. At 8;400 ft he issued a Traffic Alert. We had a TA; but not an RA. As we descended back to 8;000 ft the autopilot showed no intention of capturing; so I engaged ALT Hold manually. Investigation of the logbook showed a writeup of the B autopilot several days earlier; so we switched to the A system and completed the flight with no other issues. Maintenance; along with Dispatch; was notified and the B autopilot was written up and MELd. The Pilot's main focus of flying the aircraft was interrupted and I failed to monitor the automation system as was necessary. I will never trust the autopilot system to do its task unmonitored again.