I was either lacto-ovo vegetarian or vegan for over a decade, and though I was strict about meat-derived (so like, I didn't eat candies with gelatin because that is made from bones/hide), I wasn't as strict about dairy/eggs-derived, or as I put it at the time, I didn't worry about the "incidentals." Which means at home I was strictly vegan and I went for vegan when eating out, but I didn't cross examine the waitress about if the bread had eggs in it when I ate a sandwich, that kind of thing.  And now I am an omnivore again because of all my food allergies.  The funny thing is, this is obviously what my body wants - not only am I properly allergic to so many things, but my "ibs" symptoms are like 95% gone and I dropped six sizes.  I had been reading all the nutrition books, but not taken into account that every body is different.
It might be easier for you as far as people not understanding what you mean or presuming they know better, to try the term "flexitarian."  I know it's just another label, but people will (generally) understand that it means you are predominantly vegetarian but will make exceptions sometimes.