Pilot reported landing safely after descending below 500 ft AGL on approach to CMA in an attempt to stay below a forming cloud layer.

Narrative

Several hours prior to departure the TAF for my destination CMA was showing the field going VFR to IFR at the time of my arrival due to low cloud; an updated TAF issued about an hour prior to my departure pushed the forecasted IFR conditions to 2 hours after my estimated arrival time. On descent into CMA the METAR was giving 9 sm visibility; sky clear. As I was descending through approximately 2;000' MSL in clear air CMA Tower advised me to keep my speed up and make right traffic for RWY 26; adding that the field was about to go IFR soon but that 'I should be able to make it in'. I saw a thin cloud bank in front of and below me; which I assumed was the reason for the Tower's comment. I saw clear sky below the base of the cloud bank which I estimated to be at approximately 1;000' AGL; and I decided to descend under the cloud bank and try to proceed to the runway while maintaining VMC.After a few seconds of flying under the base of the cloud bank I realized its base continued to descend lower and that the visibility was much less than the 9 sm I had been expecting. I had to descend further to remain clear of the clouds; and within another few seconds realized I was now flying much lower and closer to the ground than I was comfortable with or should be; I recall seeing 560' MSL on my altimeter.I decided the only safe corrective action was to initiate a climb back to a safer altitude of at least 1;000' AGL; even if it meant temporarily entering IMC conditions (my airplane is IFR equipped; I am IFR current and proficient). Initiated the climb and advised Tower I had to climb due to the clouds. I was in IMC conditions for I estimate less than 5 seconds before breaking out of the cloud layer. Tower advised an easterly heading which I followed; and after another approximately 20 seconds the cloud layer dissipated completely; I could see the airport and runway; and I joined a right base for an uneventful visual landing. Conditions on base and final appeared to me as approximately 6-10sm visibility; sky clear.I made multiple human performance errors:1) expectation bias and complacency: I considered that the TAF trend showed the weather to be improving from previous forecast; and I did not expect to have any trouble getting in VFR. This was reinforced by the METAR stating 9sm SKC.I did not consider that this METAR was almost an hour old when I was listening to it on descent into CMA; and that the weather could have changed substantially since the METAR was cut2) my initial judgement of the base of the cloud layer as 1;000' was poor; in reality I think it was several hundred feet lower. I think I wanted the base to be 1;000'; and told myself it was without evidence.3) Once I realized I had descended to an unsafe low altitude above the ground on the order of 500' AGL; it took me several seconds to decide to climb up through the thin cloud layer; and to accept that I would have to temporarily enter VMC without an IFR clearance. This hesitation on my part prolonged my exposure to the dangerous low altitude flight regime unnecessarily. It is interesting to me that if this had been an instrument approach and I had reached DA without required visual references I would have imitated a missed approach without hesitation; in this case flying VFR; I found myself surprisingly reluctant to climb into IMC even though it was clearly the less bad option compared to continuing to attempt to duck under the lowering cloud layer4) I failed to use all resources at my disposal. The first thing I should have done; while still at 2;000' in clear air; was ask Tower for more information on the weather situation as they saw it from the tower. I believe Tower would have been able to tell me that they saw a cloud bank to the North East -the direction I was arriving from- but that only a mile or so further east the cloud bank cleared; and that the approach area for RWY 26 and the runway itself was sky clear. If I had asked for and received this information; I never would have made the decision to descend under the cloud layer; instead I would have maintained altitude and proceeded east in VMC until I had the runway in sight5) I conducted poor meteorological analysis. In addition to not considering that the METAR I was relying on was nearly an hour old; I did not consider the broader meteorological picture well enough; and did not consider that this low level cloud developing late in the evening was likely coming in from the ocean; and as such that the approach end to RWY 26 would likely be more clear than the direction I was arriving from (closer to the ocean). If I had considered this; I would have been less likely to make the decision to try to descend under the cloud layer; instead I likely would have decided to maintain altitude in the expectation that the cloud would dissipate the further East I flew; as was the case in reality.After more than 3;000 hours flying time this experience was a sobering wake up call; particularly the expectation bias; complacency and then delayed decision to climb that I exhibited. The outcome of flying low to the ground in deteriorating visibility could have turned out much worse.