Since he doesn't know that Peter knows about the Skyline key, there is no reason for him to be worried about this. After all, if Peter doesn't know, the only thing he sees is Neal visiting different places in New York.