A TBM pilot reported during initial climb they temporarily lost trim control. They reset the auto pilot and regained full control.

Narrative

I was on a flight from ZZZ to ZZZ1. I departed VFR from ZZZ; intending to pick up my IFR clearance in flight. Preflight; startup; taxi; and takeoff roll were normal. Checklists were followed as is standard procedure. Takeoff was normal; plan was for a right downwind departure with a climb to 11;000 feet preset. After gear up; flaps up; and commencing the right turn; control felt heavy in the turn. The autopilot did not accept yaw damper engagement. No error was shown; it just didn't engage when YD button was pressed. Control continued to feel heavy; and because yaw damper didn't engage; rudder trim remained in the takeoff setting. I attempted to correct rudder trim manually. I am not certain whether I was correcting rudder; or may have unintentionally used the aileron trim toggle instead. I can't be certain.I continued the climb to about 3500 feet; trying to figure out why my trims were out of whack. Control was very heavy. At this point; both rudder and aileron were out of normal; and I responded by holding the AP Disconnect button to retard any potential further trim motion. At this point I called ZZZ Approach and [requested priority handling]; due to concern that I would be preoccupied with regaining full control and unable to properly avoid traffic or airspace. My intent if I could not sort out the problem would be a return to ZZZ and a difficult landing. Approach accepted my declaration; and I worked to re-establish full control. I recentered rudder and aileron trim; and reset the autopilot (powering it off then back on). During this activity; I tried to remain at around 3500 ft MSL; while remaining in the same general location. After the autopilot reset and trim resets; it appeared that all systems returned to behaving normally. I do not know whether this was a system issue or a pilot issue. I suspect a pilot induced issue. It is possible I inadvertently engaged aileron trim during takeoff; because that toggle switch is in close proximity to my torque lever controlling hand during takeoff. Regardless; while the controls were very heavy; I never lost full control of the plane; and was able to reset the trims and autopilot. Once the plane was stable and operating well on autopilot; I cancelled [priority handling] with ZZZ Approach; and requested my IFR clearance to continue the flight to ZZZ1. I'm very thankful for the great help ZZZ Approach provided while I sorted my problem.