I would argue that this is actually an advantage of story points. You don't know who will be working on it at estimation time. But a 2-pointer should take any developer about twice as long as it would take the same developer to do a 1-pointer. If you have many super fast, very senior developers you get more points done and have a higher velocity. That's where the speed will be reflected and you can see how much your team can get doen in one iteration. This allows you to plan your iteration without having to care who works on any given story.