Yeah, latency is more an issue to do with recording than playback, unles we are dealing with DSP plugins where it takes time to process the sound. In this situation, the sound coming out of the plugin can lag the playing audio. fortumately Logic can correct this if you visit the Audio > Generla preferences and set the "Plug-in delay compensation" to All ... see if that has an effect.
As Jacqueslacouth points out, you can play with the buffer size to adjust latency during recording.
What is more likely to be happening is logic is having a hard time streaming the data off the disc. This can happen if a disc becomes fragmented, or if you you have a little RAM and lots of loops playing, because your computer will be forced to use a lot more virtual memory. There's a diswcussion about this in the reason forum, about half way down this thread:
Sometimes, just running your software updater can cure these problems, as it "optimizes" your drive after updating the software.