Air carrier pilot reported there was a fumes event after Engine #2 was started during taxi for departure and after returning to the gate; the reporter noticed the flight attendants were experiencing symptoms.

Narrative

Cabin crew reported a fume event after engine 2 was started during taxi for departure. The odor was described as engine oil; strong and persistent; with the aft cabin mostly affected.A few minutes earlier; ECAM ENG2 OIL FILTER CLOGGED message was triggered several times after engine 2 was started. Abnormally high oil pressure of 180 psi was noticed by the flight crew. We advised ATC of our mechanical issue. We were instructed to proceed onto Taxiway 1 and hold short of Taxiway 2. Maintenance Control was contacted. We were advised to shut down engine 2 and return to the gate for further inspection.As we prepared to taxi back to the terminal; the cabin crew notified us of the fume event. The A320/A321 guide was referenced. The cabin crew notified us that the odor was dissipating a few minutes later.After customers had deplaned; two FAs appeared visibly ill. Their symptoms included significantly irritated eyes; nose; throat and light-headedness. Their facial expression was very different than what I had seen during the cabin crew brief an hour earlier.The aircraft was taken out of service; and our flight was swapped into another aircraft within the hour. However; the two FAs continued to appear ill. I spoke directly to a Chief Pilot to express my strong concerns that they were not well to fly. The entire cabin crew was removed.