Flight Instructor reported taking evasive action to avoid a NMAC with another aircraft in the ATC controlled traffic pattern.

Narrative

We had a near miss at Long Beach Airport (LGB) on departure from 26R.We departed 26R and were approved for a right downwind departure. We were not given any altitude restriction. Our aircraft was Aircraft X. Aircraft Y was given instructions to cross midfield at or above 2000 feet to enter right traffic for 26R. As we were turning crosswind to downwind at approximately 1600 feet the Tower gave us a traffic alert and northbound heading to fly to avoid the traffic. We were not told to stop our climb nor were we given an altitude restriction; so we continued our climb as there was airspace up ahead at 2;500 feet that we were trying to avoid. As we continued climbing; we immediately turned to our given heading. Once on our heading we noticed a much faster Aircraft Y flying directly overhead of us with only about 100 feet of vertical distance and no more than 50 feet of horizontal distance. The aircraft had deviated from their instructions from Tower and their flight radar track showed they had prematurely started their descent to pattern altitude while simultaneously drifted over the departure end of the runway and continued straight towards us in the downwind. The controller immediately told us to stop our climb and told the other aircraft to arrest their descent. We took evasive action by pushing the nose down and starting a rapid descent as soon as we saw the other aircraft. I believe three things contributed to this event. One was the controllers lack of awareness to how fast the other aircraft was flying and also that Aircraft Y had drifted towards the departure end of the runway instead of crossing over midfield as they were instructed to do. Second was Aircraft Y drifting off course and not correcting or noticing and beginning their descent prematurely without being instructor by the Tower Controller. The final factor was myself and my student being distracted and focusing on clearing the upcoming Delta airspace that we were rapidly approaching.