No, I’m not worried.
There is no difference between having a digital ID, and a physical ID like a passport or driver’s license. It’s often more privacy respecting to use the digital signature, when compared to showing someone your passport.
You can think of the digital ID as a government issued PGP keys, it’s really all it is. You can use the key to prove your identity, and apply legally binding digital signatures to digital documents.