I say, “No!” I would advise you to not remove subscribers due to email campaign inactivity.
You have worked hard to acquire those subscribers. Simply because someone hasn’t opened your email campaigns in, let’s say, 90 days doesn’t mean you should drop them like a hot potato. You never know when a non-engaged subscriber will decide to open your email and request your service/product. Unless you are experiencing deliverability challenges, I would not drop them.
However, it doesn’t mean you do nothing – you certainly need to pay close attention to subscribers who are waning in engagement, especially if they had demonstrated high levels of engagement in the past.
So, how about altering your mailing behavior? Apparently what you are doing currently is not working for a subset of your audience, so it would be beneficial to examine them more closely to determine if there is a meaningful way in which you can motivate them to engage/ re-engage with you. You could segment them into a separate list(s), then try out different delivery times, subject lines, etc.
On a smaller scale, you could even contact some folks using another means (phone, social media, etc.) and ask for direct feedback. It’s another opportunity to build relationships and stay connected.