"I would rather die than give up x" when someone hears about my dietary restrictions that allow me to feel well enough to leave the house.

Does anyone else feel like this is very ableist under the radar? Someone would rather die than live with my limitations? It's usually about something silly like dairy or mangos, but it can still really hurt. Implies that my limits make my life not worth living. Would they rather die than be in a wheel chair? I feel like maybe they are trying to be sympathetic about my experiences but it almost feels like they are blaming me for not toughing it up and eating stuff even though it will make me feel terrible, and shows a total obliviousness to the intensity of my symptoms that I deal with every day and the very real possibility of someone killing themselves due to chronic pain, limitation, and illness.