Is this a recent change or can I just not see older deleted questions because they have been deleted?
A bit of both, I guess. It's also depending on your definition of 'recent'. First thing: The Roomba deletes closed questions with a score of zero or less that aren't improved after 9 days. Moderators sometimes clean up some more, mostly to avoid having too many broken windows around. Some users won't notice a question is already closed and will ask similar ones, so cleaning them up hopefully helps a bit. But this does mean the question page will likely not show many closed questions older than those 9 days.
If you're thinking of longer terms, like years, there have been a few other things that make automatic cleanup by the Roomba easier (less voting, less quick answering) and a developing scope since the site reached public beta that means questions that could be asked in 2017/2018 would end up closed and deleted these days, while back then they reached HNQ and got inflated scores.
Why is this happening? Do almost all new users have fundamental misunderstanding of what is on-topic here or is moderation too strict?
I would say there's a pretty big learning curve to posting on IPS. There are quite a few questions that are just asking us 'what should I do'. Even the people that do seem to understand what is on-topic don't seem to know that their questions should focus on the interacting part. Take a look at the current close reason that's mostly used for questions: It's for needing details or clarity. Often, people either don't include a clear goal, don't clarify what they mean by 'handle the situation', or tell us all about the other person but nothing about the specific interaction they need help with!
Are there any plans to attempt to change this (via stronger hand holding of new users, more lenient rules regarding close votes etc.).
Yes, if you look at the side-bar, currently it features a post on rewriting the help/on-topic page to provide clearer hand-holding. The previous post I linked you was likewise featured. Other ideas include rewriting the good questions faq and linking to it from help/on-topic, and eventually adding clearer close-reasons so there's less commenting needed and eventually looking at things like this when the help center and meta posts are in order. Hopefully, this will guide new users better in including the necessary details in their questions.
More lenient rules sound nice to drive traffic right now, and I wish we could! But if we allow the wrong kind of questions to thrive now, we'll later have to explain to users again that they are no longer allowed to skip those parts that are so necessary to make a good question. That's less than ideal.
I'm asking this mainly because this site covers topics that I am personally interested in but in the current state I am afraid to ask a question because I have the feeling that if it's even slightly flawed it will be shot down immediately and frustrated with the extremely low rate of "answerable" questions being posted (i.e. the already low number of new questions minus those that are immediately closed).
Well, for the first part I have a solution for you: You can post a draft of your question in our Sandbox, where people will be able to
shoot it down work with you on anything you might've missed before you post on main. We can tell you if the question can work on IPS, or whether there is something you left out or if you included details focusing on the 'wrong' things. Don't be afraid to use it!
As for being frustrated at the low level questions that are open for answering, I get that. I too wish there were more questions in a state that could be left open for answers... so again, please use the Sandbox and add one!