Are classic back rollers too big for the Westfjords?

Hi folks!

First time poster, amateur bikepacker here.

I'm planning a trip around the Westfjords of Iceland for this summer (following the bikepacking.com route), and am wondering if two Ortlieb classic back-rollers are too big for the terrain (40L capacity b/w the two of them).

I don't really want to use saddlebags as I like the convenience of panniers, so I'm wondering if it's worth investing into smaller bags, such as ortlieb's gravel panniers.

For what it's worth, I'll also have a half-frame bag and a smallish 6L bar bag. The rest of my kit (tent, sleeping system) is pretty dialed and lightweight.

Have you cycled the westfjords or other relatively challenging routes with ortlieb's classic back rollers? Would handling be a serious problem on bumpy gravel descents? Would the weight/ handling be a bother on the considerable uphills?

Thanks a bunch in advance!