Beechcraft V35 pilot reported a near miss with an ultralight aircraft while entering the traffic pattern at a non-towered airport. The pilot maneuvered to avoid a collision then had another conflict with the same aircraft on final approach; resulting in a go-around to avoid the ultralight.

Narrative

Approaching ZZZ from the south; I made a radio call 7 miles south inbound for landing XX. There was a Pacer on the radio entering a 45-degree downwind XX at that time. I began slowing and announced when I was 2 miles south of the field. I was flying a VNAV path to pattern altitude as published. The pacer was on the base leg and mentioned geese on the runway. From previous experience; I knew the geese like to sit on the approach end of XX and are sometimes stubborn to get off. I called that I was entering a downwind and saw no traffic other than the pacer that was close to turning final. I clicked off the autopilot to fly the overhead crosswind entry to the downwind. I looked down at the field to see a flock of geese on the approach end of XX. When I looked back up; I saw an ultralight higher than me flying on a downwind heading. I took evasive action by increasing bank angle to about 60 degrees to avoid them; but we got very close. Close enough that had I not seen them; we would have impacted. I then joined the downwind; being the only prudent move I could make. I then announced and flew base; and on turning final; I saw the ultralight was turning inside of us to land. I flew final to an early go-around. I called that I was going around for the ultralight. I made another lap in the pattern and landed uneventfully. On the ground; the pilot of the ultralight was teaching a student and walked away from the student to ask why we were trying to make a point as to get so close to him. I told him I didn't see him and I made a radio call at 7 miles; another at 2 miles; and another turning downwind. His first response was that he didn't need a radio; and he then stated that his radio had been broken for some time and he was going to fix it. I asked if he had ADSB; he said it was not installed. He then proceeded to tell me that a midfield crosswind entry is not acceptable at this airport and that I was unfamiliar with the airport due to noise abatement. I told him I was aware of the noise abatement procedures for runway XY; that there were no published procedures for XX; and that the noise-sensitive community south of the field I did not overfly them.Being so close to ZZZ1 and inside the mode C veil; it was my understanding that he needed at least ADSB out capability. But he may fall under an exemption; I'm not sure.The next day; I approached airport management; they indicated they had had previous issues with this individual and strongly recommended that he operate with a radio.The fact that he is teaching a student to operate an ultralight into congested airspace with no radio or ADSB is not setting a good example of best practice and is turning out new pilots that will do the same thing. If the other aircraft had a working radio or ADSB; that would have greatly increased situational awareness for all parties. I believe this incident would never have happened had either piece of equipment been working or installed. I was scanning for traffic in the area but failed to pick them up while also looking for the pacer traffic and the bird activity