A Decathlon pilot reported engine vibration; loss of power and burning smell on initial climb resulting in a return to the departure airport.

Narrative

Was picking up Person B who had dropped off an aircraft for annual service at ZZZ airport. Flight from ZZZ1 airport to ZZZ airport was uneventful and airplane ran normally. Shut down and waited approx. 35 minutes for their arrival in the aircraft. Plan was to fly to ZZZ2 to drop them off and then head back to ZZZ1 airport. Discussed who would fly and Person B requested I fly because I was more current in type. Normal start. Normal run up. Weather was light wind (240 at 8kts) and CAVU. Departed runway XX with normal power and airspeed. My routine is to keep full takeoff power until above 500ft then reduce throttle to 25' MP (Manifold Pressure) and then reduce prop RPMs to 2500 to achieve 25 squared climb to cruise altitude (this day would have been 4500 with flight following to ZZZ2). Upon slowly pulling back the throttle to reduce the MP I saw that the MP gauge was not responding and remained at 27'. I continued to pull back the throttle to see if it would change to diagnose the problem and notified Person B in the back of the situation. To eliminate the possibility of a broken throttle cable; Person B used their throttle control to further reduce the throttle to see if there was any response. During that attempt the power suddenly dropped and the MP needle suddenly dropped from 27' to 12'. We noticed that one of the cylinder EGTs was higher than it should have been on initial climb - instead of around 1250 it was 1450 which is more typical of cruise after leaning. I put the throttle back in and immediately advised ZZZ Tower that we were coming back to the field in the event the situation was deteriorating and engine power was unreliable. With Tower's permission I turned into the downwind. At this time the power stabilized at around 20' MP and it seemed to be stable at 1000ft. I advised Tower that we wanted to troubleshoot the problem a bit before landing and Tower gave us permission to maneuver to the east of the airport. I turned east and advised Person B that I was going to apply takeoff power for a bit and then ease it back to 25' again to simulate the situation to see if we had the same reaction. As soon as I put in full power there were several very loud bangs; the whole cowling shuddered heavily and began constantly shaking; and the MP and engine power dropped to idle. I pulled the throttle to idle and immediately turned to the runway and [requested priority handling] with Tower. The aircraft cowling continued to shake heavily with loud sounds and a burnt smell was in the cockpit. Person B and I immediately split CRM tasks. I focused on establishing glide speed; assessing our rate of descent; and navigating to the runway; while Person B used their controls in the back to adjust the alternate air source and the throttle to see if they could get any power for the glide. Person B was successful in smoothing out the engine shaking and getting around 1000rpm of power; which ensured sufficient speed for the glide to the runway. I took the aircraft on a perpendicular path directly to the aim point boxes of runway XX; kicked it around to align with the runway; brought the remaining power to idle; applied a brief slip to lose the extra 50ft I carried into the approach; and landed the aircraft smoothly at a full stall three point landing. Upon landing the aircraft again began to shake and thump as if both main gear were flat (they were not). I was able to coast off the runway onto taxiway 1 and reapply around 600rpm to continue the roll into the ramp where I shut it down. After getting out and inspecting the aircraft there was nothing observable wrong that we could see. We checked the oil dipstick and it was 6qts and clear. We decided to perform a test start to see what the engine would do. I started the aircraft while Person B and a ramper observed from the outside. Immediately upon starting the aircraft made three loud backfire pop sounds and had a different deeper engine noise than normal so I immediately shut it down again. We called another person in another aircraft to come and pick us up and take us back to ZZZ1 airport. We have spoken to our local mechanic and but Person B was unable to determine what occurred without inspecting it. A mechanic in ZZZ airport will inspect it tomorrow to provide us with more information on what caused the partial engine failure. I suspect it was either a defective fuel servo or an induction leak; but only the mechanic can tell us. The aircraft engine completed a full OH (Overhaul) with nickel plating of the pistons to reduce corrosion and had a total of 65 hours SMOH and is regularly flown about 1-2 times per week. There were no prior indications of any engine problems and it ran extremely well. There was insufficient time during the event to perform any additional checklists or evaluations to determine potential causation and my focus was on getting the aircraft safely on the ground as soon as practicable in case of fire or uncontained engine failure.