This isn't convincing. The "I am a man" does modify the "I like cheese" even if that grammatical modifier "when" isn't there. The "I" must be a man to like the cheese. That "I" can't be a mouse. So you have modified your second clause by including your first clause.
Now you have just left sentence structure and grammar and everything else behind.
If your 9th grade English teacher told you that you were wrong on the modifier, would it convince you? (Is that still when you learn this stuff?) I would probably concede the point if mine did, even though mine wasn't the brightest bulb.
We need a grammer nazi in here, I'm not sure "modifier" is even the correct term for this.