I'm trying to sign in with my phone but it won't work.

Phone sign-in only works after your phone number is verified. Sign in with your email first, go to your Account page, and verify your phone there. After that you can use phone codes to sign in.