Cessna 400 pilot reported passenger door departed inflight.

Narrative

After performing preflight checklist and run-up; instructed from Tower and taxied to Runway 35L. Lifted off; weathervaned into wind coming from NE. At approx 2500 ft AGL and 1000 feet past the end of Runway (35L/17R); the passenger door opened; slowed aircraft such that passenger could secure the door but door had been torn from aircraft. Turned crosswind; then on downwind leg advised ZZZ Tower that I had lost a door and requested to return to the airport; requested to land on 35R where I landed without incident.The front passenger has flown with me several times and I saw the passenger turn door handle and lean against the door when asked if doors were secure. I did not double check the passenger door and I was unable to visualize the handle as the passenger has a fairly large body habitus. I was bringing some tools that trapped my safety belt latch and I struggled to free the belt prior to taxiing - this struggle contributed to a feeling of urgency prior to departure. The passenger and I both thought that the door was completely latched but in retrospect realize that the door was only partially latched.The passenger door has always been problematic/ difficult to fully close. The fractured door landed in a [Company] parking lot approx 0.5nm North of airfield; where the door landed harmlessly in a parking lot - did not strike any objects or people. I did feel rushed as I wanted to complete my flight from ZZZ to ZZZ1 prior to sunset - but performed checklist twice as I did feel rushed.