Both my tires are like this, should I be worried ?

It is fine, most wheels develop some wobble at some point, that is very minor. If it gets worse you might want to do something about it.

Also, is it the rim that is doing it or just the tyre? Some tyres wobble even ln straight rim.