An interesting question.

I know I am much more comfortable giving my filter a clean if I think I see a spec or smudge on it. I am much more reluctant to give my front element a clean unless it really needs it.

I actually gave this a test a couple of afternoons ago when I was taking a photo toward the late afternoon sun and getting bad lens flare. I tried removing the filter and was no better. To some extent that makes me feel less worried about the impact of a filter on lens flare. I have a mid level hoya filter.

I'm quite carefully with my gear, but still have some marks on the filter. Eventhough it doesnt show in any images, it makes me happy I had a filter on the lens or the marks might be on the front element.