There were some teasers at the end of 4.10 that make me think that it won't be all hearts and flowers between Neal and James/Sam...so I hope that the story will play out with Neal being firmly in Peter's camp (and at his side, if not in his bed).

But yeah - think what might have happened to Neal if Peter hadn't been the one to catch him!