@MAB made a very good point with a statement that we need to define what 'classic theme' means because some people thought that it means plain minifig heads, only bricks and plates, etc. For me, all that is vintage. Classical means originating from Lego, not from IPs like Ferrari, Marvel, etc. Classical means 'original' Lego stuff that lasts (for me, at least), so not Atlantis or Power Miners. Classics are Castle, Space and Pirates, at least.
IP-based themes are a riskless choice for Lego. And it works; it's bringing them more sales because people are buying branded Lego themes. 'Oh, that would be so cool from Lego' is the anthem of nowadays. Until that source of new sales dries up, I don't see any reason to make more classical themes. Don't change what works.
Lego has changed, but kids are still able to completely disassemble the Millennium Falcon and build an ugly giant something, so despite more non-interchangeable parts (wedge plates, pointy stuff, Technic panels, etc.), creativity hasn't been affected at all with IP sets. It was always more about the building experience than about empowering creativity.
Despite the fact that IP themes don't seem to be original, from the building perspective, they're stunning. With the current combinatorial explosion, it's true art to make good stuff from Lego. Even AI is not able to build these objects in Lego. There is innovation, but in a different place than we would look for it – it's deeply hidden in building techniques, mainly due to SNOT (Studs Not On Top) and hinge elements and mixing Technic with system bricks.
@Mister Phes and @Khargeust made interesting points about AFOL market share. This is a market with huge growth potential that will undoubtedly shape Lego's way. I would bet that they made Galaxy Explorer 10497 just because of AFOLs. Modern kids don't care about classic themes, but AFOLs can bring them back.
So, to sum it up, classic themes are dead for kids because IPs work better. Classic themes are being squeezed out also because of a limited number of Lego themes and the need to constantly change the portfolio. But growing AFOL customers can bring it back.