Pilot flying PA-28 aircraft reported engine malfunction and fuel exhaustion during cross country flight.

Narrative

Both tanks were filled to capacity on Day 0 after arriving at ZZZ1. The FBO helped me add fuel and observed both tanks completely full. After adding fuel; I never started the engine; the plane was towed into a parking spot by the FBO with a tow bar. Before departing the next day (Day 1); as part of the preflight; I drained fuel from both tanks. When pouring the fuel that was drained back into each tank; I verified both were completely full as they were the prior day. There was no rain and no water observed after being parked outside overnight.I filed IFR from ZZZ1 to ZZZ with routing ZZZ3 ZZZ4 ZZZZZ ZZZ5 ZZZ6 direct. The requested and assigned altitude was 7000. The alternate was ZZZ2. Enroute time was calculated to be 3 hours 32 minutes. I added 20 minutes to the alternate (29 NM) plus a 45 minute reserve. Thus the total fuel required was 4 hours 37 minutes. Endurance was calculated at 5 hours 20 minutes.Without starting the engine; I called ZZZ7 clearance on my cell phone and received my clearance. After filling both tanks the prior day; I would estimate the engine was run for a total of about 5 to 10 minutes prior to takeoff. Except for the short run up at 2000 RPM; the engine was run at about 1000 to 1200 RPM to taxi to the runway. ZZZ1 has no control tower so I immediately taxied to the runway where I finished the takeoff checklist including the engine run up. When I contacted ZZZ7 clearance; they immediately released me for departure. There was no traffic in the pattern and I immediately took off on Runway XX.The aircraft does not have a fuel meter which shows the actual fuel burned. The enroute altitude was 7000 ft. and I aggressively leaned the fuel mixture. After climbing to the assigned altitude; I kept the engine at 2500 RPM. I did not go 'mixture rich' until I encountered the first known engine problem over ZZZ2.Shortly after ZZZ3; a layer of clouds developed below me. Perhaps another 30 minutes later; I began to go in and out of IMC. At about that time; I heard an unidentified bang. A short time later; I heard a similar bang. All gauges showed everything running normally. I was wearing a noise canceling headset. I 'wrote it off' as something moving around in the back of the aircraft. After passing south of the ZZZ8 area; I remained IMC for virtually the remainder of the flight. The rain was heavy at times although the air remained relatively smooth with only a few bumps.Shortly after passing the ZZZZZ waypoint; both the GPS ground speed and the indicated airspeed dropped off by 10 to 25 kts. and at one point by as much as 35 kts. This continued for the remainder of the flight. Thus; the flight time continued to increase. I attributed the ground speed decrease to the bad weather; an increasing head wind and the low pressure area off the coast. I knew that was eating into my fuel reserve; but I still believed I had enough fuel. Although that didn't explain the reduced indicated airspeed; I wrote it off as well to the weather. Looking back in hindsight; I think the two bangs were engine related and the decrease in indicated airspeed was most likely that the engine was not developing normal power shortly after we passed the ZZZZZ waypoint.As I approached ZZZ2 at 7000 ft.; I was told to descend to 5000. Conditions were solid IMC at night in heavy rain. I powered back from 2500 to 2000 RPM. The engine immediately began running very rough and sounded like it was going to quit. So I [advised ATC] at that point. I was able to keep the engine running smooth again by moving the power back to 2500 RPM. Thus; I decided to keep the engine at 2500 RPM through the descent for the remainder of the flight since this was the only RPM I could find where the engine would run smoothly.When I [advised ATC]; according to the gas gauges; both tanks had nearly 10 gallons of fuel each. I [had] between 10 and 20 gallons on board. According to the gauges; engine oil pressure; fuel pressure and temperature appeared normal.Although I knew I was near the ZZZ2 airport and I could have chosen to land there; I felt attempting a visual approach would have most likely been a death sentence. I had no idea if; when and where I would break out of IMC conditions. When I lost the engine; the vacuum warning light came on and I assume I have no attitude indicator. Without doing an ILS approach; I didn't want to take a chance that I could find the airport visually especially considering the terrain below. Mentally I figured that would add about 10 miles out and 10 miles back if I wanted to do the ZZZ2 ILS while ZZZ was only 29 miles. I was already set up for the ILS XX approach into ZZZ. I knew it was a 29 NM 'straight in' approach from my current position. I was at 7000 ft. and I believed the weather was better at ZZZ.I believed that the engine would continue to run if I kept it at 2500 RPM so I decided to descend; keeping that engine RPM and keeping the indicated airspeed no higher than the top of the white arc operating range. ATIS information Sierra for ZZZ said ceiling 3500 so I knew I would be able to see the runway. I navigated direct to ZZZZZ1; the ILS XX IAF.As I broke through the last layer of clouds; I was able to see the lights on the ground. The engine quit on approach several times. With the exception of the last time; I was able to get it restarted each time. I pushed the engine mixture to full rich at the point where I experienced the first engine trouble. I verified several times that mixture was full rich. I went to full throttle and then back a little. I turned on the electric booster fuel pump. I tried carburetor heat. I believe I only switched fuel tanks one time because at no point did either fuel tank ever read empty. Although I don't expect a fuel gauge to be accurate; I do expect it to be accurate at both full and empty. Since neither fuel gauge ever read empty; I always assumed I had some fuel.I lost vacuum several times as a result of the engine quitting. As I got closer to the field; it appeared the DG was no longer accurate possibly due to the loss of vacuum. I decided to use the GPS DTK and TRK with little reliance on the DG. I asked ATC to confirm my heading and asked for any course correction.Once I determined I was not going to make the runway; I turned slightly to the right of the runway center line so that I would land on the grass. I assumed that by trying to line up directly with the runway; I could run over the runway edge lights which might cause a fire in addition to doing heavy damage to the plane.We were met by Fire Personnel. They asked us if we had any injuries. Neither of us did. There was no damage to the aircraft or any property. The Fire Personnel observed some slicks of oil on the cowling. The first was from the flap where you normally add oil. The second was near the bottom of the cowling on the right side. This is the first time in my aviation experience that I ever saw oil slicks on the top of the cowling. When I visually checked both tanks using a flashlight; it appears both were empty. Never throughout the duration of the flight did either fuel gauge ever indicate empty.