Everything on their terms

I am the non-ADHD spouse and my husband was recently diagnosed with ADHD. We've been together 13 years, married for 8. He finally sought a diagnosis after a few years of impulsive decisions that had a increasing impacts, financially and emotionally, on our family and I was at the end of my rope. He is trying meds and in therapy. I have decided to stick with it with the hope things can improve. This forum has been so enlightening for me and so grateful I found it. So many things said here I could have written myself. 

Among the many ways our relationship suffers is this sense that things are constantly on his terms, and it's been getting worse. If I ask if he wants to do something with me - either in the now or sometime in the future - he deflects, avoids answering, or sometimes it seems like a look of panic crosses his face for a moment. I'm not talking about big or difficult asks. I'm asking about things like sitting together for a few minutes on a quiet evening, seeing a movie, having dinner together (either out or one that I've cooked), listening to the outdoor concert a block away, going for a walk, intimacy, etc. I have tried being clear about what I want but that doesn't help. As a result, I rarely even bother to ask anything anymore. Forget about asking big things, I long ago learned not to bother planning trips or buying tickets to events in advance. As I write this it is a beautiful evening, the kids aren't home, and he's on some mission to accomplish a task that has been sitting for a month but is suddenly urgent. I'm lonely. 

But then.... fast forward to a few days after I've asked about 'x', and he'll ask me if I want to do that 'x' and gets angry and withdrawn if I say no. Early on I would have said yes because I figured I should take what I can get. For a while now though, I feel resentful that things have to be on his terms and say no. If I go along with it I just feel sad and angry the whole time. If I try to explain why I'm saying no, he doesn't, or says he doesn't, remember what happened. This same dynamic plays out trying to find a time to have meaningful talks about our relationship. He'll walk in a room and start talking and I feel blindsided. I explain that I would prefer to plan a time so we, or at least I, can both be clear and focused. and he deflects. I'm not sure how we'll be able to work on improving our relationship and finding ways to navigate marriage and family with his ADHD if we can't just be together let alone make time to really communicate. I find myself wondering why I should stay if I'm basically living alone plus managing the household because he hasn't been a partner in a long time. It's like this weird cycle where to fix one thing we have to fix another.

I'd so appreciate knowing whether non-ADHD spouses have felt this way and/or get some perspectives about what's going on in his head.