Air carrier Captain reported an excessive bank angle when correcting a turn from a heading deviation while climbing on a departure procedure. The pilot recovered the bank angle and followed ATC instructions and continued the flight.

Narrative

(The following incident occurred over approximately 1 minute and 30 seconds after departure from Newark).We were cleared for the Newark5 Departure to 2500 and to expect vectors on course. Departures at that time were from runway 22R intersection M. The FO and I reviewed the plate and discussed the threats because neither of us regularly operate out of Newark. We talked about the notes about avoiding LNAV on the company pages. Then I briefed the departure and explained that it would be hand flown using Heading Mode; we built the box accordingly with the appropriate frequency and distance ring. Performance data gave flaps 5 and due to the gusty winds we elected to conduct the max thrust takeoff procedure. Everything between the gate; taxi and takeoff was normal.We were cleared for takeoff at approximately XA:36Z. At 400' I began the left turn to 190 and we were handed off to Newark Departure (119.200). We checked in and the instructions we received were to 'turn left heading three-zero-zero; climb and maintain one-zero-thousand'. FO readback the instructions; spun the heading bug left to 300 and set the correct altitude. I verified each. At the same time we were beginning the cleanup phase but I noted to the FO that although I wasn't certain this instruction seemed non -standard so I intended to temporarily keep the flaps at 1 and the speed at 210 to help keep the turn radius tight and the climb steep. My thought was that perhaps they were planning to have us cross over the field at 10;000 similar to how LAX occasionally handles vectored departures; but I don't operate out of Newark often so I wasn't certain about this. Either way I didn't want the turn radius to be wide. This will be very relevant later.It took about 30 seconds for Newark Departure to come back on the radio and ask us why we were going left and to command an immediate right turn to heading 270 (no longer to 300; although later we would be asked to 'please keep turning' as if we just stopped out of non-compliance). This was a major startle for us in the flight deck; we were in a steep climb to the left and being told to immediately bank right. My FO spun the heading bug and I executed the turn. My gut sank out; adrenaline spiked and I became very nervous that we somehow deviated from ATC's instructions. I didn't realize it in the moment but my FO caught it and called out for me to watch my bank which was then around 45 degrees to the right. I believe the startle combined with the spike in stress of a possible deviation; loss of traffic separation or airspace excursion contributed to my over reaction. The aircraft gave the 'bank angle' call two different times during this period; as it rightly should have. A second controller (possibly a supervisor) then came over the radio and commanded us to stop our climb 'now!' and I leveled off; rolled wings level and asked the FO to set 3000' in the altitude window and hit level change. For a moment we tried to re-cage our flight deck and collect ourselves from the startle that just occurred. We had started to accelerate towards 250 knots when I noticed the speed tape and realized the flaps were still at 1 and called for 'flaps up after takeoff checklist'. I have no reason to believe we exceeded a flap speed; the speed bug never went past 250 from what either of us saw and we never heard an aural alert indicating an overspeed. It was just something that had been lost from our attention in the commotion but we finally got back to it before it became a problem.Between the two controllers we were eventually vectored back to 300 and out of their airspace. I believe the departure controllers realized they had perhaps given the wrong initial instruction and therefore never followed up with us on the issue. Either way it became very troubling for us in the flight deck because we spent the next 4 hours debriefing ourselves about our mistakes: Did we go the wrong way?What had we heard or thought we had heard?What willwe do in the future?Why did I over control the aircraft?What are the passengers thinking?How about the flight attendants?It was very embarrassing. I spoke with the flight attendants a little while later in the flight; explained what had occurred and asked if there were any injuries or concerned passengers and they said it was a little exciting for about thirty seconds but no; there were no injuries and no passengers seemed to really be concerned.