I've had a similar problem. I've set up a project for live use, where i've got physical instrument ports routed directly to the channel strip of the instrument that it is controlling.
This works for my purposes, since i'm playing live and not recording, but i can't record any instruments that are configured this way. I'll explain what i've done though, because it should get you half way there.
You need to go to the Clicks and Ports page in the environment and disconnect the SUM port from the Physical Input (SUM is all midi ports combined). Then you can cable individual ports to individual channel strips. You might need to either
a) cut and paste the channel strips from the mixer into the clicks and ports page or
b) put the physical input object in the mixer page
in order to cable them up
Another thing to note is that the ports on the physical input are assigned seemingly arbitrary names (my padkontrol is called Port A, and my x-station port doesn't even have a name, it's just a blank space). You'll need to just try connecting each port to a channel strip to determine which is which.
You'll notice that this allows the controllers to play specific instruments, but they won't record. I think you need to patch the output of the channel strips into something else, but i don't know what (sequencer input doesn't work). If you figure it out please tell me because i can't figure this part out myself!