I don't think there's anything ethically wrong with it, so long as it is private land and you have an "in" with the owner - that's their privilege, and while I am not happy about private land that the owner used to leave open for recreation, it's their train set, and if I have to walk in to play on it, that's better than them declaring it off limits.
Seems there are gates going up on public lands, which is kind of puzzling how that works. Not happy about that.
That said, I think you are setting yourself up for a confrontation.