Is it normal to hate comics and manga for this reason?

I don't read comics/manga very often, only really if a tv show or movie I watch has a comic/manga tie-in. But when I do read them, I often find it difficult to tell which character is supposed to be talking.

Obviously if a speech bubble points directly to a character, then it's clear that that character is the one talking. But often with a panel, the characters will be off-screen, yet will still be talking, and so the speech bubbles don't point to anyone (or sometimes the characters won't even be off-screen, but annoyingly the speech bubbles still don't point directly to anyone, they just hover around the characters). So you just have to tell from context which speech bubbles belong to which character. In these scenarios, they're supposed to make it obvious which characters are saying which lines, but I usually find it difficult to figure out.

Sometimes I'd only figure it out by the time I read at least half the dialogue in the panel, by which point I'd realise that I had assigned the speech bubbles to the wrong characters, and so I had been reading it completely wrong. Then I'd feel like I have to start over because I had been imagining the wrong characters saying the different lines. Even if I realise at that point that I had been reading it correctly, the uncertainty I had while reading it is still really annoying.

Sometimes I will skim through all the dialogue until I find some clue as to who is supposed to be speaking in that speech bubble, so that I can work out which characters are supposed to be saying all the other speech bubbles, so that then I'll be able to read the entire panel with assurance that I'm reading it correctly. But then doing this can spoil the dialogue.

So all of this really spoils the whole reading experience for me. I have to either read while stressing that I'm reading it incorrectly, or I have to skip ahead (and risk seeing spoilers) in order to figure out which speech bubbles belong to which characters, so that I can be sure that I'm reading it correctly.

And then sometimes, even after reading the entire panel thoroughly, I wouldn't figure out at all who is supposed to be saying each line, the scene would be just as consistent if each character's dialogue was swapped. It's only when I reach the next panel that I can figure out who is supposed to be talking in this panel. Or I may not even figure it out at all.

It's so frustrating, especially since one of the reasons the characters would be off-screen while talking is for dramatic effect. Like for example, they might show a desolated city and a load of dead bodies while the survivor characters talk about it from safety off-screen, and so it is a really cool effect. But not knowing who is supposed to be saying which lines just ruins it for me. I can't enjoy it, and it just ruins my enjoyment of comics as a whole.

Is this just me, or do other people have this problem? I wish each speech bubble was just labelled with the character's name. That would eliminate any ambiguity.