A Center Controller reported a TBM9 departure did not contact them in a timely manner and stopped their climb below the Minimum Vectoring Altitude.

Narrative

ZZZ Tower called for a release on TBM Aircraft X off runway XX to ZZZ. My release was issued as runway heading with no altitude stated because if no altitude is stated it is covered in our LOA with ZZZ Tower that they will assign the flight plan altitude or 140 whichever is lower. I had just issued a traffic call to a VFR that had traffic that was a conflict for them when the TBM departed. I noticed the TBM was level at 040 for 3-4 radar hits (4.5 sec sweep). The TBM never checked on frequency so I tried them in the blind and they responded that they were level at 040. I immediately climbed them to 140. Then after giving up the sector a few moments later I told my supervisor about the aircraft and they said that ' I shouldn't worry about it' and that I responded quickly to prevent further issue. It seems that tower missed the readback on the departure clearance and because the aircraft never checked on frequency I could not verify their assigned altitude. In my professional opinion rather than waste time taking the 15-20 seconds to issue a Local Airport Advisory (LAA) issuing a climb to a safe altitude is more important in this scenario. I don't know I wouldn't do anything different.