This is absolutely theoretical and wrong. If there is a possibility of your device getting seized, you don’t a luxury of not knowing the technical issues. At least, somebody should make a good setup and you need to have a good awareness. For countries without free speech, it’s generally difficult to buy a SIM card without id. If you can, buy two. One for signal, one for ordinary use. If they arrest you and you are important figure, they can get information with drugs or torture. So, this is not doing a good threat modelling, but making wrong assumptions.
There are hundreds of different attacks to get data from the physical device. Besides, Session does not have PERFECT SECRECY. Just take your recovery code and login in with another device. You will see the messages sent in the last week. That’s far more worse than exposing your number.