Flight instructor on test flight for C172 with new engine reported engine failure inflight. Returned to airport and landed uneventfully.

Narrative

Engine Test run; first flight with new engine. Maintenance with CFI onboard. Departed RW X at ZZZ As instructed by the Maintenance Company stated 'Pull engine RPM's back to 2300 after takeoff; cruise climb out; to not put a lot of stress on the prop and new engine.'Person A was on the Right seat; and Person B was in the pilot postion. Person B is a Private licensed pilot and has flown with Person A numerous times; and had done a handful of XC flights around AZ. Person B was flight and Person A was supervising. Upon takeoff right at 300 FT AGL; Person B as instructed by Maintenance Company; pulled back the engine RMP's from full power to 2300 and continued straight out without any anomalies RW X at ZZZ.Another second or two later; the plane has a violent shake 'Transient' left and right shake. I immediately looked at Person B and went for the mixture knob to increase it. Mixture at time of failure was at the top of the green Arch 10gals +.I witnessed Person B increase the mixture which fixed the situation for a moment or two; then the violent shake occurred again. Person A immediately took controls and banked the aircraft to the Right as previously discussed with Person B prior to the start of the flight. Person A called the ZZZ Tower advising (ZZZ Tower turning back right now!)ZZZ tower replied with 'Are you declaring?' Person A's response was 'Yes declaring lost engine on takeoff!'Some of that transmission back to the tower at ZZZ was blocked by transmission from another aircraft. Immediately upon the second violent shake Person A took the mixture knob back to cease output of the engine; and hopefully stop the violent shake that was occurring. This deliberate action worked and stopped the engine noise; shaking; and Person A was able to get the vehicle Aircraft X back to the opposite runway YY.After coming to a stop on the high-speed taxiway at ZZZ Person A called tower/ground and stated the need for help getting back to the FBO.Person B was given permission and drove his vehicle to tow (Aircraft X) back to his repair facility.