Strange sounds on the lowest gears of the new cassette

Recently i changed the whole transmission on my bike, and began to hear strange metal friction sounds on the lowest gears of the new cassette after tuning the rear derailleur. I think that it is coming from chain rubbing against the nearest gear. If i try to adjust the cable tension to get rid of these noises, i loose the upper gears (the tension becomes too loose and the 10th gear on the shifter actually corresponds with the 11th gear). I've bought a new hanger but it didn't resolve the issue. Now I think that the derailleur itself might be bent, or that the shifter is flawed (releases the cable unevenly). it is also worth to add that before replacing the transmission I had a similar problem, but the chain would try to jump the the bigger gear instead of just rubbing against it slightly.

Here are the things i changed on the bike:

Replaced the cassette from SLX 11-40 to m8000 XT 11-46
Removed the front derailleur
Replaced the 36-26 chainring with an m8000 34t XT
Replaced the old chain with a HG701
Generally serviced the bike.

My rear derailleur remained unchanged (rd m8000 deore XT SGS), which I run without a B-link (it wasn't there when the bike was originally bought). The b-gap is fine being ~7mm on the lowest gear. The video shows the gears change and the possible source of the sound.

Open to any suggestions