Government UAS pilot reported flying outside of their approved COA altitudes.

Narrative

I was flying a large UAS. The aircraft was flying as an approved participant in an exercise in the R-XXXX complex and was launched and recovered out of ZZZ airfield in the R-XXXY complex. The COA gives us routing between the 2 restricted areas on an IFR flight plan in Class A airspace. I became the Pilot Flying. I picked up my IFR clearance to return to the R-XXXY complex. Usually I talk with military ATC and they give me the clearance as filed back to R-XXXZ under our 'Alternate' routing. But; this time I was in a slightly different location and at FL290. Military ATC handed me off to the nearby ZZZ Center to get my IFR clearance. ZZZ Center cleared me direct to ZZZ VOR at FL220. I exited R-XXXA and was flying direct to ZZZ VOR which is our 'Primary' COA approved routing back to the area from R-XXXX.The ATC controller told me to descend down to FL120 and handed me off to Military Arrival. They stepped me down to 8;000 ft. MSL. The workload during descent was keeping me really busy. I didn't remember our COA altitude restriction is between FL180 and FL230 while en route between Restricted airspace. I instinctively followed the ATC altitude clearances down. I'm supposed to tell ATC unable if they give me a clearance above or below the approved COA altitudes. I entering the R-XXXY complex with approval from the controlling agency; then landed at ZZZ Airfield.After landing the aircraft; I called my Flight Operations Manager to tell him about being directed by ATC to descend below the COA routing altitudes and failing to tell them unable since they are likely unfamiliar with the COA restrictions for our operation.