Captain reported loss of aircraft control while conducting a post maintenance functional check. The flight crew disconnected the autopilot and returned to departure airport.

Narrative

This was a post maintenance flight for Autopilot and Yaw Damper repairs. During climb out at approx 1;800 ft.; with a reduced power setting for level off the AP was engaged. The aircraft was in a moderate RIGHT bank to track the departure. Pilot Flying requested Pilot Monitoring turn on the Auto Pilot. Almost immediately the aircraft rolled LEFT past the horizon and nosed DOWN excessively. AP/YD was disconnected and aircraft hand flown. Pilot Flying requested immediate return to ZZZ for a visual landing to XXL and Pilot Monitoring begin landing checks. Maintainers on-board came forward to observe and were immediately told to return to seats for expedited landing. Pilot Flying kept aircraft under 180 for control ability and began descending turn towards downwind. In hindsight Pilot Flying(me) should have requested Pilot Monitoring [request priority handling]. We were given an climb to 2;000 from 1;500 ft. and then cleared for the visual. A flight control issue was then reported but we did not ask for assistance. During final Pilot Flying asked for a wind check (it was under 7 kts.) this was due to the 1/2 to 3/4 rudder required to keep the nose centered. Aircraft landed without incident and was returned to maintainers. The flight was 6 minutes. I should have had us [request priority handling]. I was near mental capacity flying the aircraft and helping a new Pilot Monitoring complete his tasks to change us from departing to landing.