I have now again tried readjusting them. They're non-conforming now as well. Some places i can't get the seeker in between, but the drums also have a small "dip" in the middle due to wear.
After adjusting them now, the wheels turn nicely around, and the car rolls very well on flat surface, indicating that the brakes are not contacting the drums when not breaking, at least not that much.
One thing that does worry me, is that the brakes do get hot. Took a trip without braking at all, just using the hand brake, and the brakes do got hot. It's not like they're super super hot, but enough that it's uncomfortable touching them.
Is that a concern i should do something about?
I tried loosening the bands so that they made no contact at all, but then due to the non-conforming bands, the car didn't brake at all.
It seems that they do have to be slightly in touch with the drums to get braking.