Hi Lisa. Welcome!
Your first paragraph are my sentiments exactly.
Arguments are good, quarrels are not:
In the words of G.K.Chesterton: Never allow a quarrel to interrupt a good argument

Anyone who uses the term "going to Hell" when speaking of your situation is plain out flat wrong. That is NOT Catholic teaching. So not only are they are wront, they are themselves sinning in being judgmental.
That being said, though, if your husband wants to be in full communion with the Church, you can help by seeking an annulment.
A valid marriage is for life, as we read in Scripture - and considering the hardships that come with or after some marriages, it can become an admittedly painful and difficult issue to recognize that. That's what annulments are for: To look close to see if the marriage was valid.
Might you and your husband consider speaking to his Pastor about seeking an annulment?