Rust is a new language. Its designers are close to solving the problems they set out to solve, but they haven't fully succeeded yet. Heavy use of "unsafe" indicates that there are things you can't properly express in the language, or can only express inefficiently.
It's worthwhile to separate the two. If you can't do it at all, there's a problem with expressiveness. If you can't do it fast, there's a problem with performance. It's useful to try to do things without using "unsafe" to cheat the system. Then you find out what the language needs to do faster. That's a trouble spot to be identified. There may be an optimization which leads to a safe and fast solution. Or there may be a way to make something checkable so that it's not necessary to do something unsafe.
It's worthwhile to separate the two. If you can't do it at all, there's a problem with expressiveness. If you can't do it fast, there's a problem with performance. It's useful to try to do things without using "unsafe" to cheat the system. Then you find out what the language needs to do faster. That's a trouble spot to be identified. There may be an optimization which leads to a safe and fast solution. Or there may be a way to make something checkable so that it's not necessary to do something unsafe.