Air carrier Captain reported a loss of control of aircraft control after starting the #1 engine due to not setting the parking brake when the aircraft disconnected from the tug.

Narrative

From Gate XX; we were cleared to 'pushback to East Line; Spot XX'. Due to the close proximity and angle of this spot; compared to the gate; the pushback commenced slowly; and continued at a slow and safe pace; set by the Tug Driver. The tug was mostly out of view during the pushback procedure; due to its smaller size; and our normal seat configurations. The pushback was followed by a Pulling Forward to the exact spot that we were cleared to: 'Spot XX'. While the Push-Back Tug-Driver was pulling us forward to 'Spot XX' he cleared us to 'Start Engines'. During Engine #1 Start; the First Officer and I were monitoring Ramp Frequency on Com 2; Ground Frequency on Com 1 (since our push spot was very close to the 'ground area' & departure runway); as well as communicating/listening to the Tug Driver...so 3 'frequencies' were being monitored simultaneously. Around this time; either the Tug Driver failed to tell us to 'Set the parking brake' (when he was done pulling us forward to 'Spot 1'); or we missed hearing him tell us to 'Set the parking brake'; as we were monitoring multiple 'frequencies'. Regardless of either scenario; I did Not say 'Parking Brake is Set (verifying with the 'ON' brake illumination); Cleared to Disconnect; Thanks for the push'...like I verbatim tell every Tug-Driver after the Parking Brake has been Set; and Cross-checked. After Engine #1 was started; the First Officer noticed the Push-back Tug driving away off to his Right-hand side. I did not initially see this though. Right after this; we both saw the pushback wing-walker...who transitioned to the nose of our aircraft...moving his wands in an unfamiliar fashion. At this moment we realized he was trying to get our attention; by potentially signally the 'Set your parking brake - On' Hand Signal...except he had 2 wands in his hands still. While trying to understand what he was telling us we noticed the aircraft start to roll forward; towards the wing-walker with wands (at the nose of the aircraft)...towards the perpendicular service road / ground taxi area. At this moment I slowly applied the Toe brakes; coming to a complete stop again; and setting the parking brake.The starting of engine #1 made it so the aircraft crept forward on the 'east line'; further than where we suspect the push-back Tug Driver initially wanted us...the tug driver who out of the vicinity at this time.After we set the parking brake; we received the 'area clear Salute' from the aircraft nose 'wing-walker'; confirmed the Tug-driver was gone; and double-checked the area to make sure all was set to continue. At this time we started our 2nd engine; and continued normally.At all times we stayed in our cleared area; given by Ramp Control: 'East Line; Spot 1'; stayed clear of the service road; and continued our procedures.Monitoring 3 'frequencies' simultaneously could have been a factor. There could also have been a 'Monitoring breakdown'; and/or failure to hear. A Clear and Concise 'Parking Brake is On; Cleared to Disconnect; Thanks for the Push' must be made at all times (which was not said initially; due to not hearing the 'set brakes' command by the tug driver.I think all Tug Drivers and I should have an Increased awareness; by adding additional language / confirmation that the brake has been set; and that we Actually heard each other. Using the term 'Roger' more can potentially help this; like all transmissions by ATC have while flying in Location A. I will be using this term more. I will also try to increase my awareness by monitoring closer; especially in transitional moments of movement...such as the tug driver slowing / stopping the aircraft.