So I am out on a walk doing usual thing photographing building angles. I admit I did climb 2 steps off the boulevard to get a better shot of a building. I then proceeded to step off the steps and shot from the public boulevard sidewalk. When I was on the public boulevard/sidewalk a man (not security) came up to me and said "Excuse me...do you have a permit to photograph this building? I repled nicely "No I don't...should I?" to which he said yes as this was private property. I apologized and said I did not know, that I this as simply a hobby and I thought there was no issue as I was on the public boulevard. If he confronted me on the steps I could see the problem as I was trespassing on the property. however,....there were no signs posted saying I couldn't photograph the building I was photographing it from the outside and when he confronted me I was on the public sidewalk shooting up not in.

I am curious to know who was right in this situation. Once he confonted me I stopped shooting and did not want any hassle so I left and I have no problem with it.
However, but I am curious to know if in fact I could challenge him (I wouldn't...better to just smile, apologize and leave quietly I think rather than start something unless he was being a jerk about it...which he wasn't) on my rights as a photographer to shoot a building from the public sidewalk.
Do I indeed have to have a permit to photograph a building from the sidewalk..could he challenge me if I was on the other side of the street?
If there are no signs posted can I shoot the building from public property?
Anybody have any answers by all means share...I tried looking at the
Rights found here, but it is not clear...
Ambientlight.ca - Laws