This is a great question, Mikeal - I would imagine that it's common (not just in you and me!) to debate the merits of the holidays and spending them around family when being around them doesn't necessarily help us to feel good about ourselves.
My question for you is that aside from the things that you feel bad about in the holidays, is there anything about which you feel good when you leave your family at the end of them? It sounds like you worry that you'll miss out on a positive that you may have yet to see if you don't go. There sure is always that risk, but if this is a consistent pattern in your family that has been going on for a long time, I would encourage you to think about whether or not you really believe this pattern is going to change. If you're in counseling and trying actively to reconcile these feelings in yourself, there might be reason to believe that things might change, even if it is only in the way you think about them and not necessarily a change in their behavior.
However, if you already know all of the answers to these questions, and you know that you don't want to go the holidays with your family and you are just looking for support and validation to make a change in this time of year so that you have an easier time, I would definitely encourage you giving yourself the space to do that, if that's where you are. Your mental health is important. If stopping going to the holidays with your family will even give you an inch in preserving it and you feel ok about doing it, go for it! Worst case scenario, if you hate not going to the holidays with your family, there's always the following year.
Nothing has to be permanent about a decision like this if you don't want it to be.
I hope this helps!