Pilot reported NMAC with another aircraft while crossing through the airport traffic pattern.

Narrative

I was flying the second lap of my banner route and I was running behind. Because of this I chose to cross the south side to ZZZ airspace rather than go around the peninsula. I knew ZZZ was using the XX runways and they gave me permission and said remain 2 miles from the airport. I have done this transit many times in the past but never when they were using the XX runways. Also; I am usually asked to remain 1 mile south of the airport. As I am climbing and about to turn onshore; I realize I need to be higher because the terrain slopes up from the airport to the south. So; I made a 180 deg turn to about 040 while offshore to gain altitude. When I reached 1300 feet I turned onshore and as I did I saw a plane below and to my left making a downwind to base turn and I realized I could be near the path of the downwind traffic so I continued climb and my turn to head south. ZZZ Tower then asked I saw another aircraft on downwind and when told where to look I reported them in sight and noted that they were below me and would pass behind me. ZZZ then reminded me I was supposed to remain 2 mi south of the airport. At the time of the near miss; I was still 3 miles from the airport and from the flight data I believe I did remain about 2 miles south of the airport.I do think I was not as careful when turning onshore as I know I did not check in the direction of the turn before I turned. That may have been due to my concern with checking the altimeter to make sure I was high enough to come onshore and being in a rush to get to the next leg of my flight. I should have been on high alert knowing the downwind traffic would be headed in my direction.