If milk or milk products affect you, it is not because milk is bad for you. The milk you use is bad for you.
Use milk that has not been homogenised. The process of homogenising milk breaks the molecules, which your body recognises and knows how to us, into something that cannot be identified or used and therefore is usually stored in your fat.
I buy farm milk, using some of it to make my own butter, feta, cream cheese and full cream yoghourt (with honey and sunflower seeds). Having my milk full cream, with a nice head of it at the top, has not caused me to put on even half a kilo.
I eat a lot of lentils, with uncooked slices of onion and black olives placed in it, with grape or apple vinegar and olive oil. However, I also love my thick steaks and lamb chops. As for rare roast beef, a delight, but not something I have often, as it is not convenient for a single person. No potatoes, just sweet potatoes.
One or two days a week, I only eat salad, but it has lots of extras, like big raisons, olives, celery (lowers blood pressure), a hard-boiled egg (eat one if you have had too much sugar, it helps level out your 'high') and anything else I think would make it 'interesting', so that it is not exactly the same each time. Boredom and predictability is the worst thing for me, when to do with eating food.