huh... you didn't mention effects ;-))
But seriously, sometimes it does. It depends on your workflow... I might record and arrange and mix somewhat simultaneously... other times, recording takes precedence and then I'll be mixing and adding effects later down the line.
The I/O Buffer size you can use would vary, as you mention, per project and per system. I know many users who are on an I/O setting of 64...
Have you tried enabling the Low Latency Mode?
(Should be in the Transport...)
It will limit the latency caused by plugins and bypass plugins that are causing too large a delay.
Also, worth bearing in mind to steer clear (if possible) of latency inducing plugins, like Reverbs, delays, limiters... But with Low Latency Mode turned on you should be good to go!