Maximise the video and watch the left hand HUD pane from 0:10 to 0:11.
The dotted black line coming from the front of the car (which I am assuming is the intended route) quickly snaps from straight ahead, to a hair pin right, to a normal right turn.
Ignoring the fact that the right turn happened to be into a pedestrian crossing with people on it - what was the car even trying to do? The sat-nav shows it should have just continued forwards.
I am astounded that software capable of these outputs is allowed on the roads. When could crossing a junction then taking a hairpin right back across it, ever be the correct thing to do?
I've made this argument here a few times and am always shot down, but I think its important to highlight that the airline industry has an extremely robust history of automation and HCI in critical transportation scenarios and it seems to me that all the lessons that we have learned have been chucked out the window with self-driving cars. Being able to effectively reason about what the automation is doing is such an important part of why these technologies have been so successful in flight, and examples like this illustrate how far off we are to something like that in cars. The issue of response time, too, is one we cant ignore, and it is certainly a far greater challenge in automobiles.
I don't have answers, but it does seem to me like we are not placing a premium enough on structuring this tech to optimize driver supervision over the driving behavior. Granted, the whole point is to one day NOT HAVE to supervise it all, but at this rate we're going to kill a lot of people until we get there.
Hi, aerospace software engineer and flight instructor here. I think you get shot down because the problems just aren't comparable. While I agree that there may be some philosophical transfer from aircraft automation, the environments are so radically different that it's difficult to imagine any substantial technological transfer.
Aircraft operate in an extremely controlled environment that is almost embarrassingly simple from an automation perspective. Almost everything is a straight line and the algorithms are intro control theory stuff. Lateral nav gets no more complicated than the difference between a great circle and a rhumb line.
The collision avoidance systems are cooperative and punt altogether on anything that isn't the ground or another transponder-equipped airplane. The software amounts to little more than "extract reported altitude from transponder reply, if abs(other altitude - my altitude) < threshold, warn pilot and/or set vertical speed." It's a very long way from a machine learning system that has to identify literally any object in a scene filled with potentially thousands of targets. There's very little to worry about running into in the sky, and minimum safe altitudes are already mapped out for pretty much the entire world.
Any remaining risk is managed by centralized control and certification, which just isn't going to happen for cars. We aren't going to live in a world where every street has to be (the equivalent of) an FAA certified airport with controls to remove any uncertainty about what the vehicle will encounter when it gets there. Nor are we going to create a centralized traffic control system that provides guarantees you won't collide with other vehicles on a predetermined route.
So it's just a completely different world with completely different requirements. Are there things the aerospace world could teach other fields? Yeah, absolutely. Aerospace is pretty darn good at quality control. But the applications themselves are worlds apart.
I'm actually in complete agreement! What sticks out to me is your assessment that the flight environment is "embarrassingly simple from an automation perspective", which I agree as well (as compared to cars). And yet despite that simplicity and decades at it, we still run it with an incredible robust infrastructure to have a human oversee the tech. We have super robust procedures for checking and cross-checking the automation, defined minimus and tolerances for when the automation needs to cease to operate the aircraft, training solely focused on operating the automation etc. But with cars, we somehow are super comfortable with cars severely altering behavior in a split-second, super poor driver insight or feedback on the automation, no training at all, with a human behind the wheel who in every marketing material known to man has been encouraged to trust the system far more than the tech (or law), would ever have you prudently do.
I'm with you that they are super different, and that the auto case is likely much, much harder. But I see that and can't help but think that the path we should be following here is one with a much greater and healthy skepticism (and far greater human agency) in this automation journey than we are currently thinking is needed.
I agree completely. It's a very difficult problem from a technical perspective, and from a systems perspective, we've got untrained operators who can't even stay off their phones in a non-self-driving car. (Not high-horsing it here; I'm as guilty of this as anyone.) Frankly I'll be amazed if anyone can get this to actually work without significant changes to the total system. Right now self-driving car folks are working in isolation - they're only working on the car - and I just don't think it's going to happen until everyone else in the system gets involved.
> we still run it with an incredible robust infrastructure to have a human oversee the tech
Airplanes are responsible for 200-300+ lives at a time, so it’s quite incomparable to road vehicles. Of course it makes sense to have human oversight in case something goes wrong.
On the flip side, the average car driver is not very skilled nor equipped to deal with most surprises - hence the ever present danger of road traffic.
I’m not sure why AI drivers are held to such insanely high standards.
The claim that self-driving cars are being held-up to a higher standard than human drivers is simply false. Self-driving cars so far have a far worse record than the average of human drivers, which is remarkably good. Human accidents are measure in term of "per million miles driven". Self-driving cars have driven a tiny total distance compared to all the miles human drivers have driven.
I’m reminded of GWB’s “soft bigotry of low expectations”. (Even if you didn’t like him, it remains apt.)
Sorry, most drivers I’ve seen avoid the kid that follows the bouncing ball into the street and all the other random events. A few drivers are crap. But justifying automation based on that few is lazy and incorrect. Elon is just full of his own crap on Tesla’s autopilot.
> most drivers I’ve seen avoid the kid that follows the bouncing ball into the street and all the other random events
I've seen many poor drivers over the decades, but can't think of any human drivers who would reliably and repeatably crash into a stationary fire truck in their lane. "Full Self Driving" vehicles, on the other hand...
> Airplanes are responsible for 200-300+ lives at a time, so it’s quite incomparable to road vehicles
Your analogy is wrong. Buses can have 50 people. I would compare a bus to a medium/small airplane.
Also, it is regrettable that cars don't have the FAA-required electronics, software, or integration processes. When I read that a Jeep's braking system was compromised through its entertainment system it was apparent that the aircraft lessons had not been taken aboard by the auto industry.
> We aren't going to live in a world where every street has to be (the equivalent of) an FAA certified airport with controls to remove any uncertainty
actually ive been thinking this is exactly the win self driving vehicles have been looking for. upgrade, certify, and maintain cross country interstates like I-70 for fully autonomous, no driver vehicles like freight, mail, hell even passengers. maybe that means one lane with high-vis paint and predefined gas station stops and/or some other requirements. i bet the government could even subsidize with an infrastructure spending bill, politics notwithstanding.
there cant possibly be a problem with _predefined_ highways that is harder to solve than neighborhood and city driving with unknown configurations and changing obstacles. i feel like everyones so rabid for fully autonomous Uber that the easier wins and use cases are being overlooked.
Well, to control things, you'd have to have a highway that's only for self-driving vehicles. And then you'd need to get them there - with what, human drivers? (losing the cost savings) Maybe you could use this for self-driving trucks between freight depots.
The problem with this is - why not just use trains at this point? Trains already an economical solution for point to point transportation.
>fully autonomous, no driver vehicles like freight, mail, hell even passengers. maybe that means one lane ... and predefined gas station stops and/or some other requirements. I bet the government could even subsidize with an infrastructure spending bill, politics notwithstanding.
Yup, that's a train. I really do wish US rail hadn't been turned into what amounts to private property. I live along the Amtrak Cardinal line and would love to use it to travel. But the low speed and frequent stops for higher-priority freight mean a trip takes longer than driving and usually costs more than I would pay in gas.
> We aren't going to live in a world where every street has to be (the equivalent of) an FAA certified airport with controls to remove any uncertainty about what the vehicle will encounter when it gets there.
I do sometimes wish we'd just devote one lane of our larger freeways to self-driving cars exclusively. You could let the cars drive as fast as they want, and charge per-mile tolls for any car that uses it when the road is congested. Ideally, you'd charge based on occupancy as well: an empty car should have to pay more than a car with a human in it, and a car with 2+ people might be allowed to use the lane for free.
Cars are vastly more complex to do navigation for than planes too so we need to be even more careful when making auto autos. Plane autopilots are basically dealing with just the physical mechanics of flying the plane which while complex are quite predictable and modellable. All of the obstacle avoidance and collision avoidance takes place outside of autopilots through ATC and the routes are known and for most purposes completely devoid of any obstacles.
Cars have a vastly harder job because they're navigating through an environment that is orders of magnitude more complex because there are other moving objects to deal with.
Which is one reason I come back to thinking that you may see full automation in environments such as limited access highways in good weather but likely not in, say, Manhattan into the indefinite future.
Unexpected things can happen on highways but a lot fewer of them and it's not like humans driving 70 mph are great at avoiding that unexpected deer or erratic driver either.
ADDED: You'd actually think the manufacturers would prefer this from a liability perspective as well. In a busy city, pedestrians and cyclists do crazy stuff all the time (as do drivers) and it's a near certainty that FSD vehicles will get into accidents and kill people that aren't really their fault. That sort of thing is less common on a highway.
This is what I'd be happy with. Something to get me the 20-50 miles between cities (Atlanta<->Winder<->Athens in particular), or through the closed highway loops around them. Driving within them isn't so boring that my focus wanders before I notice it's wandering.
We could just expand MARTA, but the NIMBY crowd won't allow it. People are still hopped up on 1980s fearmongering about the sorts of people who live in cities and don't want them infesting their nice, quiet suburbs that still have the Sherriff posting about huge drug and gun busts.
Public transit isn't a panacea for suburbs/small cities. I'm about a 7 minute drive from the commuter rail into Boston but because of both schedule and time to take a non-express train, it's pretty much impractical to take into the city except for a 9-5 workday schedule, especially if I need to take a subway once I get into town.
For me, it's more the 3-5 hour drive, mostly on highways, to get up to northern New England.
> Which is one reason I come back to thinking that you may see full automation in environments such as limited access highways in good weather but likely not in, say, Manhattan into the indefinite future.
This is why there are SAE levels of automation and specifically Level 4 is what you're describing. Anyone claiming their system will be Level 5 is flat out lying.
Even just in the US, there are some cities that can be fairly challenging for an experienced human driver, especially one unfamiliar with them. And there are plenty of even paved mountain roads in the West which can be a bit stressful as well. And that's in good weather.
Highways are probably the best case scenario for full automation but we've seen scenarios where even that idealized environment has deadly failures in a handful of Tesla crashes.
They're still in a vastly more controlled environment than cars and moving at much lower speeds as well. If an auto taxiing plane has to avoid another airport vehicle something has gone massively wrong. Judging by this video [0] I'm not sure they're even worrying about collisions and are counting on the combination of pilots and ATC ground controllers to avoid issues while taxiing. It looks like the cameras are entirely focused on line following.
I recently avoided a startup prospect because they were looking to build a car OS that wasn't hard real-time. The very idea that they're trying to develop such things that might intermittently pause to do some garbage collection is freaking terrifying.
What has hard real time have to do with garbage collection? You can have concurrent GCs (with [sub]millisecond pauses, or no Stop-the-world at all) but you also need 'hard real time' OS to begin with. Heck, even opening files is far from real-time.
Then you need: not-blocking data strcutures, not just lock-free - that are much easier to develop. Pretty much you need forward guarantees on any data structure you'd use.
You usually need garbage collection because you are allocating in the first place. And allocating and releasing adds some non-determinism. You apparently don't know how much exactly needs to be allocated - otherwise you wouldn't have opted for the allocations and GC. That non-determinism translates to a non-determinism in CPU load, as well as to "I'm not sure whether my program can fulfill the necessary timing constrains anymore".
So I kind of would agree that producing any garbage during runtime makes it much harder to prove that a program can fulfill hard realtime guarantees.
To play devil's advocate...and because I'm just not that educated in the space, is this really a huge deal? If we're talking couple ms at a time delays, isn't that still vastly superior to what a human could achieve?
If you can set a guaranteed maximum on the delays (regardless of what those limits are), you're hard real-time by definition. The horror is that they weren't building a system that could support those guarantees.
What if say, a system is written against an indeterminate timed GC like say, Azul or Go's, but code is written in a way that proves GC times never exceed X, whether by theory or stress testing. Is this still seen as 'unguaranteed'?
It depends on your system model (e.g. do you consider memory corruption to be a valid threat), but it could be. In practice, actually establishing that proof purely in code is almost always impractical or doesn't address the full problem space. You use hardware to help out and limit the amount of code you actually have to validate.
GC pauses can be hundreds of milliseconds. You could perhaps use a particular GC scheme that guarantees you never have more than a couple millisecond pause, but then you have lots of pauses. That might have unintended consequences as well. I'm also not sure that such GCs, like golangs, can really mathematically guarantee a minimum pause time.
Hard real time garbage collectors have existed for decades. Of course you can mathematically guarantee a minimum pause time given a cap on allocation rate. What's stopping you?
You don't even need a cap on allocation rate, GC can have during allocation w/o fully blocking, it'd 'gracefully' degrade the allocation, itself. It'd be limited by CPU/memory latency and throughput.
If ernie the intern decides to use a hashmap to store a class of interesting objects as we drive, you could end up with seconds of GC collection + resizing if it grows big enough.
That doesn't seem especially bad. The car could, for instance, predict whether or not it was safe to do garbage collection. Humans do the same when they decide to look away while driving.
A human that looks around while driving is still taking in a lot of real-time input from the environment. Assuming they're not a terrible driver they examined the upcoming environment and made a prediction that at their current speed there were no turns, obstacles, or apparent dangers before looking away. If they didn't fully turn their head they can quickly bring their eyes back to the road in the middle of their task to update their situation.
If a GC has to pause the world to do its job there's none of that background processing happening while it's "looking away".
That's horrifying on such a deep level. There should be mandatory civil service for programmers, but you just get sent somewhere cold and you gotta write scene demos and motion control software for a year to get your head on straight. :P
To echo this, as someone who has done some work with formal specifications, I have to say it seems like the self-driving car folks are taking a "move fast and break things" approach across the board, which is horrifying.
The mechanism by which those lessons were learned involved many years full of tragedy and many fatalities including many famous celebrities dying in those plane crashes. Obviously, we do not want to follow that same path, but at the moment that's exactly the path we're on.
The US govt isn't going to do anything until there's a public outcry, and historically there won't be a public outcry until there's a bunch of famous victims to point to.
> The US govt isn't going to do anything until ...
I think this attitude is defeatist and absolves us of doing anything. It's a democracy; things happen because citizens act. 'The US government isn't going to do anything' as long as citizens keep saying that to each other.
> it seems to me that all the lessons that we have learned have been chucked out the window with self-driving cars.
I think it’s unfair to lump all self driving car manufacturers together.
The traditional car companies have been doing research for decades (see for example https://en.wikipedia.org/wiki/VaMP), but only slowly brought self-driving features to the market with part of the slowdown because they are aware of the human factors involved. That’s why there’s decades of research on ways to keep drivers paying attention and/or detecting that they don’t.
“Move fast and break things” isn’t their way of working.
These lessons have been chucked out the window by second tier (e.g., GM/Cruise) and third tier (e.g. Tesla and Uber) competitors, who have recognized that the only way they can hope to catch up is by gambling that what happened to Uber won't happen to them.
The car FSD - aircraft autopilot analogy is deeply flawed, and nowhere near instructive. Let's consider some details:
What aircraft autopilot does is following a pre-planned route to the T, with any changes being input by humans. The aircraft autopilot doesn't do its own detection of obstacles, nor of router markings; it follows the flight plan and reacts to conditions of the aircraft. Even when executing automatic take-off and landing, the autopilot doesn't try to detect other vehicles or obstacles - just executes the plan, safe in knowledge that there are humans actively monitoring for safety. There is always at least two humans in the loop: the pilot in command who prepared and inputed the original flight plan and also inputs any route changes when needed (collision and weather avoidance), and an air traffic controller that continuously observes flight paths of several aircrafts and is responsible for ensuring safe separation between aircraft in his zone of responsibility. Beyond that, an ATController has equal influence on all aricraft in his zone of responsibility, and in case one does something unexpected, it can equally well redirect that one or any other one in vicinity. Lastly, due to much less dense traffic, the separation between aircraft is significantly larger than between cars [1] providing time for pilots to perform evasive maneuvers - and that's in 3d space, where there are effectively two axes to evade along.
Conversely with car FSD - the system is tasked both with following the route, and also with continuously updating the route according to markings, traffic, obstacles, and any contingencies encountered. This is a significant difference in quantity from the above - the law and the technology demands one human in the loop, and that human can only really influence his own car at most. Even worse, due to density of traffic, the separation between cars is quite often on the order of seconds of travel time, making hand-over to driver a much more rapid process.
I am moderately hopeful for FSD "getting there" eventually, but at the same time I'm wary of narrative making unwarranted parallels between FSD and aircraft autopilot.
> Being able to effectively reason about what the automation is doing is such an important part of why these technologies have been so successful in flight, and examples like this illustrate how far off we are to something like that in cars.
Is that actually the case, though?
I would hope, although perhaps I'm mistaken, that the developers of the actual self-driving systems would be able to effectively reason about what's happening. For example, would a senior dev on Tesla's FSD team look at the video from the article and have an immediate intuitive guess for why the car did what it did? Or better yet, know of an existing issue that triggered the wacky behavior?
Even if not, I'd hope that vehicle logs and metrics would be enough to shed light on the issue.
I don't think I've ever seen a true expert, with access to the full suite of analytic tools and log data, publish a full post-mortem of an issue like this. I'm certain these happen internally at companies, but given how competitive and hyper-secretive the industry is, the public at large never sees them.
They certainly are trying very hard, as far as I can tell. Tesla's efforts on data collection and simulation of their algorithm are incredibly impressive. But part of why it is so necessary is that there is an opaqueness to the ML decision-making that I don't think anyone has quite effectively cracked. I do wonder, for instance, if the decision to go solely with the cameras and no LIDAR will prove to ultimately be a failure. The camera-only solution requires the ML model to accurately account for all obstacles, for example. As crude, and certainly non-human as it is, a LIDAR with super crude rules for "dont hit an actual object" would have even at this point prevented some of their more widely publicized fatal accidents which relied on the algorithm alone.
there are keys difference between automation in e.g. aircraft, and what Tesla at al are failing at,
e.g., how constrained the environment is; and what the exposure is to anomalous conditions is; and what the opportunity window usually is to turn control back over to a human.
The thing I don't understand is, we have a much more comparable environment in ground travel: federal highways.
Innumerable regressions and bugs and lapses aside, I do not understand why so much effort is being wasted on a problem which IMO obviously requires AGI to reach a threshold of safety we are collectively liable to consider reasonable; when we could be putting the same effort into the (also IMO) much more valuable and impactful case of optimizing automated traffic flow in highway travel.
Not only is the problem domain much more constrained, there is a single regulatory body, which could e.g. support and mandate coordination and federated and data sharing/emergent networking, to promote collective behavior to optimize flow in ways that humans limited-information self-interested human drivers cannot.
The benefits are legion.
And,
I would pay 10x as much to be able to cede control at the start of a 3-hour trip to LA, than to be able to get to work each morning. Though for a lot of Americans, that also is highway travel.
Not just this, why not start with the low-hanging case of highway travel, and work out from there onto low-density high-speed multi-lane well-maintained roads? Yes that means Tesla techs who live in Dublin don't get it first. Oh well...
IMO there will never be true, safe FSD in areas like my city (SF) absent something more more comparable to AGI. The problem is literally too hard and the last-20% is not amenable to brute forcing with semantically vacuous ML.
I just don't get it.
Unless we take Occam's razor, and assume it's just grift and snake oil to drive valuation and investment.
Maybe the quiet part and reason for engineering musical chairs is just what you'd think, everyone knows this is not happening; but shhhh the VC.
I'm on my third Tesla. FSD on highways has improved so much in the last 6 years. On my first Tesla, autopilot would regularly try to kill you by running you into a gore point or median (literally once per trip on my usual commute). I now can't even remember the last time I had an issue on the highway.
Anywhere else is basically a parlor trick. Yes, it sorta works, a lot of the time, but you have to monitor it so closely that it isn't really beneficial. As you point out, its going to take some serious advances (which in all likelihood are 30+ years away) for FSD to reliably work in city centers.
I think the issue you've highlighted is one of governance. There's only so much Tesla can do regarding highways. You really need the government to step in to mandate coordination of the type I think you're envisioning. And the government is pretty much guaranteed to fuck it up and adopt some dumb standard that kills all innovation after about 6 months, so it never actually becomes usable.
I think automakers will eventually figure this out themselves. As you say, there are too many benefits for this not to happen organically. Once vehicles can talk to each other, everything will change.
To be fair, it still felt like magic. My car would drive me 20 miles without me really having to do anything, other than make sure it didn't kill me at an interchange.
And I'm now trying to remember, but I think autopilot was initially free (or at least was included with every car I looked at, so it didn't seem like an extra fee). Auotpilot is now standard on all Teslas, but FSD is an extra $10k, which IMO is a joke.
Humans are ridiculously bad at overseeing something that mostly works. That’s why it is insanely more dangerous.
Also, the problem is “easy” for the general case, but the edge cases are almost singularity-requiring. The former is robot vacuum level, the latter is out of our reach for now.
is why there we haven't seen the feds stepping in via the transportation agency to develop and regulate exactly this, with appropriate attention paid to commercial, personal, and emergency vehicle travel.
The opportunities there appear boundless and the mechanisms for stimulating development equally so...
I really don't get it. Then I think about DiFi and I kind of do.
Even lower-level automation for highway driving would be super useful.
I would appreciate a simple "keep-distance-wrt-speed" function for bumper-to-bumper situations. Where worst case scenario, you rear-end a car at relatively low speeds.
I'd happily keep control over steering in this situation and just keep my foot over the brake, though lane-keep assist would probably be handy here as well. A couple radar sensors/cameras/or lidar sensors would probably be enough for basic functionality.
Disable if the steering-wheel is turned more than X degrees - maybe 20 or 25?. Disable if speed goes over X speed - maybe 15mph? Most cruise controls require a minimum speed (like 25mph) to activate.
Trying to do full driving automation, especially in a city like Seattle, is like diving into the ocean to learn how to swim.
As cool as that sounds, I'd trust incremental automation advancements much more.
I would appreciate a simple "keep-distance-wrt-speed" function for bumper-to-bumper situations.
This has been widespread for at least a decade. I'm not even aware of a mainstream auto brand that doesn't offer adaptive cruise control at this point. Every one I've used works in stop and go traffic.
The other features you want are exactly what Tesla has basically perfected in their autopilot system, and work almost exactly as you describe (not the FSD, just the standard autopilot).
I can't say that my experience agrees with this. Maybe some higher-end vehicles had it a decade ago, but it seems to be getting more popular over the past 5 years or so. I still don't see it offered on lower priced vehicles where basic cruise functionality is there, but I doubt ever will be a part of "entry level" considering the tech required.
None of the vehicles my family owns have a "stop-and-go" cruise control function - all newer than 10 years. ACC at higher speeds is available on one, but it will not auto-resume if the vehicle stops.
In this same vein cars with automated parallel/back-in-angle parking.
I think an even more 'sensor fusion' approach needs to be adopted. I think the roads need to be 'smart' or at least 'vocal'. Markers of some kind placed in the roadway to hint cars about things like speed limit, lane edge, etc. Anything that would be put on a sign that matters should be broadcast by the road locally.
Combine that with cameras/lidar/whatever for distance keeping and transient obstacle detection. Then network all the cars cooperatively to minimize further the impacts of things like traffic jams or accident re-routing. Perfect zipper merges around debris in the roadway.
Once a road is fully outfitted with the marker system, then and only then would I be comfortable with a 'full autopilot' style system. Start with the freeway/highway system, get the logistics industry on board with special lanes dedicated to them and it becomes essentially a train where any given car can just detach itself to make it's drop off.
Also, what could possibly save the most lives: simply ML the hell out of people’s faces to notice when they are getting sleepy. That’s almost trivial and should be mandatory in a few years.
A more advanced problem would be safely stopping in case the driver falls asleep/looses consciousness, eg. on a highway. that short amount of self-driving is less error-prone than the alternative.
>Unless we take Occam's razor, and assume it's just grift and snake oil to drive valuation and investment.
I think there's some of that. Some overconfidence because of the advances that have been made. General techno-optimism. And certainly a degree of their jobs depending on a belief.
I know there is a crowd of mostly young urbanites who don't want to own a car and want to be driven around. But I agree. Driving to the grocery store is not a big deal for me. Driving for hours on a highway is a pain. I would totally shell out $10K or more for a full self-driving system even if it only worked on interstates in decent weather.
Highway self driving has been around for decades[1] - and Tesla's general release autopilot can already do all that. As I understand it from ramp on to ramp off, in production vehicles, Tesla can provide an automated experience. I'm not sure how much "better" it can get.
>Highway self driving has been around for decades[1]
Driver assisted highway has been around for years... Level 5 driving requires no human attention. Huge difference.
I think what is wanted by many is level 5 on the highways. I want to sleep, watch a movie, whatever. That is much, much "better" than what we have now. Like many others, I would be most interested in full level 5 on highways and me driving in the city. That is also much easier to implement and test. The scope of the problem is greatly reduced. I think Tesla and others are wasting tremendous resources trying to get the in-city stuff working. It makes cool videos, but being able to do other activities during a 5 hour highway drive has much more value (to me at least) than riding inside a high risk video game on the way to work.
(edit) I get that I am misusing the definition of "level 5" a bit, but I think my meaning is clear. Rated for no human attention for the long highway portion of a trip.
better would be that i can legally go to sleep and let the car drive the highway portion of my trip. then i wake up and complete the final leg of the trip.
as you say, i don’t think this is entirely out of reach (even if it required specialized highway infrastructure or car to car communication). seems like lower hanging fruit than trying to get full self driving working on local/city roads.
i would pay a ton for the highway only capability…
I've watched quite a few FSD videos and in almost every single one the car barely knows what it wants. The route jumps all over the place. I'm pretty sure it's just the nature of their system using cameras and the noisy data that can generate.
The sat nav didn't update the route to go that way. The FSD system decided to. It probably completely lost track of the road ahead because of the unmarked section of the road and just locked on to the nearest road it could find, which was that right turn.
I've seen videos where it cross over unmarked road and then wants to go into the wrong lane on the other side because it saw it first. It seems like it will just panic search for a road to lock onto because the greatest contributor to their navigation algorithm is just the white lines it can follow.
If you watch the Tesla AI day presentation they explain the separation of duties that kind of explains what's happening.
The actual routing comes from something like Google Maps that is constantly evaluating driving conditions and finding the optimal route to the destination. It does this based on the vehicle's precise location but irrespective of the vehicle's velocity or time to next turn.
The actual AI driving the car is trying to find a path along a route that can change suddenly and without consideration. It's like when your passenger is navigating and says "Turn here now!" instead of giving you advanced notice.
That could be selection bias. The [edit, was '99%'] vast majority of the time the car does the boring thing are not click-worthy. Have you driven a Tesla for a reasonably long period of time?
There is a bigger lesson in there: click-driven video selection creates a very warped view of the world. The video stream is dominated by 'interesting' events, ignoring the boringly mundane that overwhelmingly dominates real life. A few recent panics come to mind.
99% perfect is not good enough, it's horrifyingly bad for a car on public roads.
When I drive, I don't spend 1 minute plowing into pedestrians for every 1 hour and 39 minutes I spend driving normally.
If a FSD Tesla spends just 99% of its time doing boring, non-click-worthy things, that is itself interesting in how dangerous it is.
To your point, I'm definitely interested in knowing how many minutes of boring driving tend to elapse between these events. The quantity of these sorts of videos that have been publicized recently gives me the impression that one of these cars would not be able to spend 100 minutes of intervention-free driving around a complex urban environment with pedestrians and construction without probable damage to either a human or an inanimate object.
99% is a an very rough colloquial estimate meaning 'the vast majority of the time' to drive the point. Could well be 99.999999%. What really matters is how it compares with human performance, I don't have data to do that comparison. The only ones that can make the comparison are Tesla, modulo believing data from a megacorp in the wake of the VW scandal.
FYI, that 99.999999 number you quoted is still bad. It means around 30 minutes of the machine actively trying to kill you or others while driving on public roads. I assumed a person driving 2 hours a day for a year.
FSD should not be allowed on the road, or if it is it should be labeled as what it really is: lane assist.
I'm not 'quoting' any numbers. I don't own a Tesla. I don't trust lane assist technology, the probability of catastrophic failure is much larger even compared with dynamic cruise control. I'll steer the wheel thank you very much. I'm not a techno-optimist, rather the contrary. I would like independent verification of the safety claims Tesla makes, or any other claims made by vendors of safety-critical devices.
What I am saying is that selection bias has exploded in the age of viral videos, and this phenomenon doesn't receive anywhere near the attention it deserves. We can't make sound judgements based on online videos, we need quality data.
I have an intersection near where I live where the Tesla cannot go thru in the right lane without wanting to end up in the left lane past the intersection (I guess it's a bit of a twisty local road). At first, it would be going in a straight line, then when it hit the moment of confusion would snap into the left lane so quickly, some 200 ms or something. Never tried it with a car in that spot fortunately. After a nav update, it now panics and drops out of auto-pilot there and has you steer into the correct lane. Nothing to do with poor visibility or anything, just a smooth input that neatly divides its "where is the straight line of this lane" ML model, or whatever.
It's actually fascinating to watch - it just clearly has no semantics of "I'm in a lane, and the lane will keep going forward, and if the paint fades for a bit or I'm goign thrur a big intersection, the lane and the line of the lane is still there."
It also doesn't seem to have any very long view of the road. I got a Tesla at the same time of training my boys to drive, and with them I'm alwys emphasizing when far away things happen that indicate you have increased uncertainty about what's going on down the road. (Why did that car 3 cars ahead break? Is that car edging towards the road going to cut infront? Is there a slowdown past the hill?) The Tesla has quick breaking reflexes but no sign of explicitly reasoning about semantic layer uncertainty
Yes, for 4 years I did, and what they're saying is absolutely true. It desperately tries to lock on to something when it loses whatever it's focused on. Diligent owners just learn those scenarios so you know when they're coming. Others may not be so lucky.
For example, short, wavy hills in the road would often crest just high enough that once it couldn't see on to the other side, it would immediately start veering into the oncoming lane. I have no idea why it happened, but it did, and it still wasn't fixed when I turned in my car in 2019. I drove on these roads constantly so I learned to just turn off AP around them, and it helped traffic on the other side was sparse, but if those weren't true, I'd only have a literal second to response.
EDIT: IMO the best thing it could do in that scenario is just continue on whatever track it was on for some length of time before its focus was lost. Because it "sees" these lines it's following go off to the right/left (such as when you crest a hill, visually the lines curve, or when the lines disappear into a R/L turn) but only in the split second before they disappear. Maybe that idea doesn't fit into their model but that was always my thought about it.
From the admittedly not much footage of Tesla FSD (1-2 hours total maybe) I’ve watched, it seems to be roughly on par with a student driver who occasionally panics for no apparent reason.
While most people see the student vehicle and give it a wide berth, there are those that see it as a "teachable" moment. We've already seen that asshats that screw with cars labeled with Student Driver just to "give 'em a lesson" are already screwing with self driving cars for the same reason: They're assholes.
Assuming by hater you mean "someone who is scared by Full Self Driving including cars veering into pedestrians", it'd help me: I'm in an urban area and don't have a car, and I'd be lying if I claimed I wasn't always scared when I see a Tesla approach when I'm in a crosswalk, I'm never sure if Full Self Driving is on or not.
Here's my hater take: "Full Self Driving" is an absolutely dangerous and egregious marketing lie. The vehicles are not full self driving and require by law a driver capable of taking over at a moments notice.
I do not believe that Tesla's technology will ever achieve "Full self driving" (levels 4 or 5, where a driver isn't required https://www.nhtsa.gov/technology-innovation/automated-vehicl...), and the labeling of their "conditional automation" system as "full self driving" is an absolute travesty.
People do honestly think they have bought a self driving car. The marketing lie absolutely tricks people.
The problem is that you could apply that argument for a drunk driver, a distressed driver, a driver without a license, a driver that hasn't slept enough, etc.
Any of those could mean a car could randomly turn into the crosswalk.
Humans are horrible at assessing risk. We worry about Teslas because it's on the headlines now, but you're infinitely more likely to get ran over by one of the cited examples than a Tesla just by sheer statistics.
Never mind the driver who can't wait the two to five seconds for the pedrestrian to cross, which is probably far more common than the above, and rather intentional.
TBH I don't think it's ever crossed my mind whether a Tesla was in FSD or not, even when driving alongside one. As long as it doesn't enter my space, I'm fine.
Walking through Manhattan, I'm more worried about the human driver nowadays than the non-human.
"I'd be lying if I claimed I wasn't always scared when I see a Tesla approach when I'm in a crosswalk, I'm never sure if Full Self Driving is on or not."
Sure - which is why I said I agreed there should be some indication of it being in FSD or not, originally.
...and it is obvious that it will actually ever get more than incrementally technically better, absent AGI.
The problem is too hard for existing tools. Good job on the 80% case; let's look for problems for which the 80% case is good enough and back away from unsupportable claims about "just a little more time" bringing FSD in environments defined by anomaly and noise, in which the cost of error is measured in human life and property damage.
As well as he/she has a professional driver paying close attention to whatever he does. Overseeing something is a really mentally trying thing and “fsd” car drivers will not be able to do that for long times.
You have it backwards. With humans driving there is a constant and unending stream of student drivers, because every new human must learn from scratch. There is no silver lining.
With self driving it takes longer for the one individual to learn, but the difference is that there is only the one individual, and it need not ever die. The learning is transferable between cars, regressions between generations need not be accepted. The problem of bad driving can finally be solved.
Given the number of different people pouring millions to billions of dollars at the problem, I think it's pretty incredible for you to be so certain that that is the case.
I mean... 5 minutes on /r/IdiotsInCars and it's very easy for me to understand how even with bizarre bugs like this, it's better than people.
People say "how could it happen?!" and get angry for a software bug - that can be fixed - and seem to accept how drunk drivers do this (and a lot worse) literally every single day. But since it's a human, that's fine!
There are pretty stringent laws against drunk driving in most places... if Tesla's "self driving" modes are that bad then they should be equally illegal
Genuinely curious: how would you even go about advancing autonomous driving without testing it in the streets?
I'm absolutely certain that they've attempted to simulate millions of hours of driving, yet this odd behavior happened in real life, which now can be studied and fixed.
If we never had the software on the wild, how would you ever really test it?
>Genuinely curious: how would you even go about advancing autonomous driving without testing it in the streets?
Genuine response:
1 don't let customers do the testing, especially if you don't train them (I mean real training about failures not PR videos and tweets and some small letter manual with disclaimers)
2 use employees, train drivers to test, have some cameras to check the driver to make sure he pays attention.
3 postpone testing until the hardware and software is good enough so you don't ignore static objects.
4 make sure you don't do monthly updates that invalidates all your previous tests.
IMO there is so much 'machine learning' in the tesla self driving system is there any way to know a bug is 'fixed' other than just running it through a probably totally boring set of tests that doesn't even approach covering all scenarios?
Yeah, I guess you could always be safer about it, but I'm really not sure it would be enough. If we substitute FSD for any software, you have code tests, QA, the developers test it, and bugs still go through. It's inevitable.
Unfortunately it's always about the incentives and on a capitalist society the only incentive is money. So even if they could be safer, they wouldn't do it unless it's more profitable, specially being a publicly traded company.
In a sense, a self-driving car might actually be easier to test for than complex software - at least parts of it.
After all, normal (complex) software tends to have lots of in depth details you need to test for; and a surface area that's pretty irregular in the sense that it's hard to do generalized testing. Some bits can be fuzz tested, but usually that's pretty hard. It's also quite hard for a generalized test to recognize failure, which is why generalized test systems need lots of clever stuff like property testing and approval testing, and even then you're likely having low coverage.
However, a self-driving car is amendable to testing in a sim. And the sim might be end-to-end, but it needn't be the only sim you use; the FSD system almost certainly has many separate components, and some of those might be easy to sim for too; e.g. if you have a perception layer you could sim just that; if you have a prediction system you might sim just that; etc.
And those sims needed be full-sim runs either; if you have actual data feeds, you might even be able to take existing runs, and the extend them with sims; just to test various scenarios while remaining fairly close to real world.
I'm sure there are tons of complexities involved; I don't mean to imply it's easy - but it's probably tractable enough that given the overall challenge, it's worth creating an absolutely excellent sim - and that's the kind of challenge we actually have tons of software experience for.
A law doesn't prevent anything, it only applies after the fact. You could argue that the prospect of being prosecuted might scare people into not doing the thing that they are not allowed to be doing, but with all the people doing the things they are not allowed to be doing anyway, I doubt a comparative legal argument helps here.
You could make a law that states that your FSD has to be at least as good as humans. That means you have the same post-problem verification but now the parallel with drunk drivers can be made.
Tesla’s “FSD” is only a little bit better than drunk drivers, whom we punish severely whenever caught, even before any accident occurs.
The fact that enforcement is patchy is irrelevant — drunk driving is deemed serious enough to be an automatic infraction.
Also, most of the time drunk drivers are not actually that bad at moment-to-moment driving. That’s why almost everyone worldwide used to do it! You can still do the basics even when reasonably drunk. That doesn’t make you safe to drive. It’s still incredibly dangerous to do.
This assumes FSD-to-drunk-driver analogy means FSD has to be a drunk driver (or a student driver as commented elsewhere) all the time, so always making the bad judgement and slow reaction like a drunk driver would.
I think that some form of responsibility has to be assigned to the FSD in some way (the manufacturer? the user? some other entity or a combination?) regardless but I haven't found any clear case of how that would work.
It also makes me wonder how we would measure or verify a human driver with intermittent drunkenness. Imagine 5 seconds out of every minute you temporarily turn into a drunk driver. That's plenty of time to cause a major accident and kill people, but on the other hand that would mean that the combination of a situation where that would happen and the right timing to not be able to judge that situation has to apply. We do of course have the luxury of not having humans constantly swapping drunk and normal driving, so it isn't a realistic scenario, but it would make for a better computer analogy.
Besides drunk drivers we also have just generally crappy drivers that just happened to get lucky when doing their driving test (although there are places where no meaningful test is required so that's a problem in itself).
I think you’ve missed my point, while adding some additional information.
- drunk drivers are also not uniformly awful drivers: they can drive OK for the most part
- they still drive unacceptably poorly
- we strictly punish them on detection, before any potential accident
- drunk drivers and FSD have more in common with each other than competent drivers and FSD
- why is FSD not held to such a preventative standard?
One can argue that FSD is like a drunk driver driving a student training car with two sets of pedals and two steering wheels, and the Tesla owner/driver is like a driving instructor. But driving instructors are trained and paid to be quite vigilant at all times. Tesla play a sleight of hand and say it’s a labor saving technology, but also you need to be able to behave like a trained and paid driving instructor... that is a conspicuous contradiction.
And I’m ignoring future FSD capabilities because while I’d be happy for it to come about, we should discuss the present situation first, and I don’t believe it’s a good example where sacrificing lives now is acceptable in order to potentially save lives in the future.
Perhaps it is lost in translation; I'm not saying the fact that someone is driving drunk only matters when an accident happens, I'm saying that right until the moment a driver decides to get drunk, the law doesn't do anything. If at the beginning of the day someone decides to start drinking and when they are drunk they get in to a car and start driving, that's when the violation occurs. Not before that like PreCrime would.
The same can't apply to FSD because it isn't consistently 'driving drunk'. That analogy doesn't hold because it is not fixed software like a GPS-based navigation aid would be. Just like humans it does have more or less fixed parameters like the amount of arms and legs you have, that doesn't tend to change depending on your intoxication.
One could make the argument that it's not as much the "haha it is just like a drunk driver zig-zagging", but the uncertainty about the reliability. If a car with some autonomous driving aid drives across an intersections just fine 99 times out of a 100, and that one time it doesn't, that doesn't mean the car software was dunk 100% of the time.
Why FSD is not held to some standard, I don't know. I suppose that depends on how it is defined by local law and how the country it is in allows or disallows its use.
The problem with prevention and detection here is that like humans, the system is not in a static state. The trained neural network might be largely the same with every release, but the context in which it operates isn't, unless the world around it stops completely in which case two trips can be identical and because the input is identical the output can also be identical. Humans do the same, even if well-rested and completely attentive, knee-jerk reactions happen.
Holding FSD to a standard of a drunk driver isn't a valid comparison due to the non-static nature of the state it is in. This isn't even FSD-specific, even lane guidance/keeping assistance and adaptive cruise control isn't static, and those are based on pretty static algorithms. Even the PID-loops used on those will deliver different results on seemingly similar scenarios.
Perhaps we should stop comparing technology to humans since they are simply not the same. The static kind isn't and neither is a NN-based one. We can still explore results or outcomes because those are the ones that have real impact. And let's not fool ourselves, humans are far less reliable in pretty much every man-machine combination. But in human-to-human contexts we factor in those unreliabilities, and with machine-to-human or machine-to-machine we seemingly don't, which is pretty much the same problem you're describing.
This will be an interesting field of development, and if we simply take death toll into account, keep in mind that for some reason seatbelts were thought to have 'two sides of the story' as well when they were first introduced and later required. As with car seats for children and infants, a good idea might start out one way and over time (with the accompanying bodycount) it gets shaped into whatever we expect of it today. Same goes for aerospace, boats and trains, and that's even without taking software into account.
> It also makes me wonder how we would measure or verify a human driver with intermittent drunkenness. Imagine 5 seconds out of every minute you temporarily turn into a drunk driver.
Certain medical conditions are analogous to this. This is handled in various ways, including doctors reporting you to the state DMV and suspending your license if your condition is not very well controlled.
The key difference is human drivers have independent software, whereas the same software powers all FSD Teslas. One human driver getting drunk/otherwise impaired doesn't affect the software of any other human driver; but if your FSD software is as good as a drunk human, then every single one of your FSD vehicles is a road hazard.
That difference is also a benefit, fix one problem, and it's fixed for every instance.
On the other hand, it's not like the software is always bad and always in the same situation. That is a big difference with a human analogy; a drunk driver taking a trip in the car is drunk for the entire trip (unless it's a very long trip etc..), so would be impaired for the entire duration.
There are plenty of people and vehicles that are road hazards and are allowed to drive (or be driven) anyway, so if we really cared about that aspect on its own we could probably do with better tests and rules in general.
What I mean is that you don't see a headline on news outlets or twitter hashtags trending for every drunk driver fatality or crash that happens. Which for me tells that collectively, we are fine with it. We accept it as ordinary, as crazy as it may be.
I regularly see "traffic collision kills [person or people]" stories in my local newspaper. I don't know if it's every one, it might be restricted to the extremely stupid (wrong-way on the highway, drunk, or excessively-high-speed) but "someone got killed" is certainly a common story in the news here.
I've certainly seen many more local "person dies in car accident" non-brand-related stories than "person dies in Tesla accident" stories.
A DUI, in California for instance, means you’re suspended for months (years if you’re a repeat offender). Since it’s basically the same “driver” driving all FSD Teslas, are you arguing that they should be suspended under the same rules for drunken behavior? If so, they will basically be out of commission indefinitely, and we won’t see headlines anymore.
No, my point is that this is overblown just because it's Tesla and people love drama, so this makes headlines and trending topics.
My comparison with drunken drivers is just with the regards to the odd behavior. If you look at a drunk driver (or even someone that fell asleep in the wheel) with a near crash, many times it would resemble this video. But the outrage from the public differs vastly.
My Audi e-tron has this habit of switching the adaptive cruise control to the on-ramp speed limit even when I’m in the middle of the freeway.
It’s something I’ve learned to deal with but the sudden attempt to break from 70 to 55 is pretty bad especially as it’s unexpected for other drivers around you.
While I’m sure Audi are much worse at updating their software to fix known issues than Tesla I find myself skeptical that the mix of hacks cars use to implement these features scale well, especially in construction zones. Hence I’m pretty content with radar based cruise control and some basic lane maintenance and then doing the rest of the driving myself.
I can imagine if my car were slightly better at some of these things I’d be a significantly worse safety driver as I’d start to be lulled into a false sense of security.
As I understood the Tesla AI presentations the path is determined using a Monte Carlo Beamsearch which looks for a feasible path while optimizing an objective that includes minimizing sideways g-forces and keeping the paths derivatives low (smooth path).
Form the videos I have seen this fails often (in that the path doesn’t go straight even if it can). Knowing a bit about randomized met heuristics myself, I am not surprised.
I think, they need to perform some post processing on these paths (a low iteration local search).
I think, they should also start with a guess (like just go straight here, or do a standard turn), and then check if the guess is ok. I think, that could help with a lot of the problems they have.
For anyone interested the Monto Carlo Tree Search used for the planning (that the tentacle shows) is described here (from the AI day video, at the 1h21m50s):
The car almost certainly decided that the road ahead was blocked. For context, this release of FSD is the first to use a new obstacle detection technique, and as a result it is the first one that doesn't drive directly into the pillars on that street without perceiving them at all. So it's very likely that this new obstacle detection system glitched out here.
The guy in the video is intentionally stress testing the system by repeatedly driving in an area he knows is poorly handled. But it's totally fair IMO, this is a real road and while it confuses humans too (as evidenced by all the people here claiming that it's illegal to turn right here), FSD must handle it better than this.
Looking at the video Tesla was probably not closer than 25 feet from any pedestrians. If pedestrians were closer I think their proximity would make the car stop as in other videos.
As to abrupt maneuver, EU limits rate of change for steering control and I think it would be wise for US to adopt something similar for driver assist systems.
Because they are sudden and unpredictable, evasive maneuvers cannot be supervised. It would be unfair to rely on driver to supervise such scenarios.
If evasive maneuver is required it would have to be done in fully autonomous way. For the time of the maneuver system would have to be responsible for driving.
>> As to abrupt maneuver, EU limits rate of change for steering control and I think it would be wise for US to adopt something similar for driver assist systems.
That's a terrible idea. First it's a bandaid over an underlying problem. Second it's a legislative answer to a technical problem, which is IMHO never a good idea.
At the end of the video (0:32 to 0:33) you can see it quickly snap to right turn again. Why is the car attempting multiple right turns while the map indicates a straight line?
I wonder if it's an upgrade bug related to calibration?
There was a video by one Tesla user over the past week that talked about how their Tesla v10 FSD software would attempt to turn into driveways, repeatedly, when driving down a straight road.
The user did a recalibration of their cameras, rebooted the car, and the problem went away.
I don't own or plan to own a Tesla, but if i did, I'm sure I wouldn't use the self driving feature because it seems more inconvenient than simply driving yourself. It's not even a driving assistant, you become the car's assistant.
Edit: to be clear, I plan to buy an EV in the future, but I'll drive it myself. Most of my driving is in a 15Km radius, urban, and babysitting the self-drive seems like more trouble than it's worth.
Speaking as someone who believed all the hype about having fully autonomous cars by 2020, I think the truth is that it is orders of magnitude harder than we thought it was and that we are orders of magnitude less capable than we thought we were.
Maybe a good interview question for experienced developers should be "tell us about a time when you attempted something in software and failed at it." In my case it was being given a 10 or 20 KLOC C server that had memory leaks all over and segfaults all over the place (written by a friend) and told to make it prod ready.
That's a rough one. Mine was fixing some physics/flight aerodynamics software that had been generated by a Fortran-to-C compiler (e.g., http://www.netlib.org/f2c/). It was in inscrutable mess and couple-years-out-of-academic-physics-turned-SE me thought I'd be able to do it easily since I had so much Fortran and C experience. I failed quite hard at that.
On the one hand I applaud Tesla for being so open about what their system is thinking with their visualisations. That could be interpreted to show a deep belief in their system's capabilities.
On the other hand, it's always terrified me how jittery any version of AutoPilot's perception of the world is. Would you let your car be driven by someone with zero object permanence, 10/20 vision and only a vague idea of the existence or properties of any object other than lane markings and number plates?
I test drove a Model 3 yesterday and this was something that really jumped out at me. I didn't try any of the automatic driving features, but driving around Brooklyn watching the way the car was perceiving the world around it did not inspire confidence at all.
Tesla's over the top marketing and hype seems at once to have been a key ingredient in their success, but also so frustrating because their product is genuinely awesome. I've long been kind of a skeptic but I could not have been more impressed with the test drive I took. It had me ready to buy in about 90 seconds. I wish there were actually-competitive competitors with similar range and power that aren't weird looking SUVs, from brands that don't lean into pure hype.
> Presumably you are aware that the visualization you see in any retail car is old software, and several iterations behind what the linked video is about (FSD Beta v10). Plus the visualization is quite a bit different (and in many ways simplified) versus what's used for the actual piloting in retail vehicles.
The extremely poor performance of the visualizations are disturbing to me. It's also completely wasted space on the display that I wish were devoted to driving directions instead of telling me what I can already see outside of the car.
I think whether it's wasted space or not depends entirely on the reliability of the system. For a many-9's system which for all intents and purposes isn't going to fail dangerously, I agree, the visualisation is just peacocking. For a beta-quality system, knowing what the car is thinking is an important driver feedback which gives additional warning before it does something really stupid.
> I applaud Tesla for being so open about what their system is thinking with their visualisations
How do you know that what is on the screen matches what the system is 'thinking'? What reason or obligation would Tesla have to engineer accurate visualizations for the real thing and show them to you (remember the definition of accuracy: correct, complete, consistent)? Would Tesla show its customers something that would make them uncomfortable or show Tesla in a bad light, or even question the excitement Tesla hopes to generate?
I think it's likely that the display is marketing, not engineering.
Presumably you are aware that the visualization you see in any retail car is old software, and several iterations behind what the linked video is about (FSD Beta v10). Plus the visualization is quite a bit different (and in many ways simplified) versus what's used for the actual piloting in retail vehicles.
And orange cones - the orange cone detection team at Tesla is outstanding. Garbage cans are also pretty accurate. Not sure why they detect garbage cans but not parked cars, but that's just me.
Computer-vision-based self-driving is a mistake. Self-driving cars should drive on their own roads with no non-self-driving vehicles. Other vehicles, traffic lights, road hazards, route changes, etc should be signals that are directly sent to the vehicles rather than the vehicles relying on seeing and identifying cars and traffic lights and road signs.
We know how to make computers navigate maps perfectly. We don't know how to make computers see perfectly. The other uncontrollable humans on the map just make it worse.
Yes this'll not work in existing cities. That's fine. That's the point. An inscrutable black box "ML model" that can be fooled by things that humans aren't fooled by should not be in a situation where it can harm humans. I as a pedestrian did not consent to being in danger of being run over by an algorithm that nobody understands and can only tweak. Build new cities that are self-driving first or even self-driving only, where signaling is set up as I wrote in the first paragraph so that a car reliably knows where to drive and how to not drive over people. Take the opportunity to fix the other problems that old cities have like thin roads and not enough walking paths.
Just to point to another possibility, since the drivers right hand is not in shot, even though his left hand is carefully poised over the steering wheel for the whole shot, it also looks like he pulled hard on the steering wheel with his right hand.
That might explain the car trying to re-route on the display, although I'm not even sure it does. To me it looks like the black line disappears into compression artefacts at just the right time.
Mostly I think we don't have enough evidence either way and speculating just expresses our desire for the technology to work or not.
Gotta give you that. He still might have moved the steering wheel with his thigh though ;-)
Crazy how it handled a shitty situation like around 6:55 in https://streamable.com/grihhc but failed there. Guess that shows what most of the ADAS engineers already know, the really hard part are those last few percentage points to make it actually reliable.
Google Maps will update (prompt first) a route in progress depending on traffic. If you don't notice right away it can be very surprising! At least it's not actually driving the car.
Tesla's system is real time, which means that it makes routing decisions on every frame, and the decisions are independent from previous decisions. It doesn't have memory. It just makes decisions based on what it actually sees with its cameras. It doesn't trust maps, like most other systems. For some reason, it thinks that it must turn right.
Situations like these are sent to Tesla to be analyzed, the corresponding data is added to the training data, and the error is corrected in the next versions. This is how the system improves.
After this situation is fixed, there will be an another edge case that the HN crowd panics over.
> It doesn't have memory. It just makes decisions based on what it actually sees with its cameras. It doesn't trust maps, like most other systems.
This is incorrect. Tesla also relies on maps for traffic signs, intersections, stop signs etc. They just don't have additional details like the others do.
> After this situation is fixed, there will be an another edge case that the HN crowd panics over.
Is anything Tesla FSD can't handle an "edge case" now? It's literally an everyday driving scenario.
You're probably right, but from your tone you seem to be implying that this is in any way an acceptable way to develop safety critical software, which is a bit baffling.
It is safe, because there's a driver who is ready to correct any mistakes. There isn't a single case where FSD Beta actually hits something or causes an accident. So, based on actual data, the current testing procedure seems to be safe.
It isn't possible to learn edge cases without a lot of training data, so I don't see any other way.
This person is not a Tesla employee being trained and paid to test this extremely dangerous piece of software that has already killed at least 11 people. This is obviously unacceptable, and no other self-driving company has taken the insane step of letting beta testers try out their barely functional software.
FSD Beta has never killed anyone. Maybe you're confusing it with Autopilot, which is different software, but also has saved much more people than killed.
I'm not saying that safety couldn't be improved, for example by disengaging more easily in situations where it isn't confident. One heuristic would be when it changes route suddenly, like in this scenario.
"In the 1st quarter, we registered one accident for every 4.19 million miles driven in which drivers had Autopilot engaged. For those driving without Autopilot but with our active safety features, we registered one accident for every 2.05 million miles driven. For those driving without Autopilot and without our active safety features, we registered one accident for every 978 thousand miles driven. By comparison, NHTSA’s most recent data shows that in the United States there is an automobile crash every 484,000 miles."
It is an inherently biased statistics, where drivers will let the car drive on parts where they are confident it can do its job, and will take over for the rare, more complex situations.
Also, the NHTSA dataset will contain old as hell cars, comparing it to a fresh out of the factory one will by itself skew the data.
Keep watching the HUD pane - before the driver takes over, it corrects and decides to go straight. In fact, it's turning the wheel to the left before the driver stops the correction.
I think this is a navigation issue. This is exactly what I would have done if I had a passenger yell "WAIT TURN RIGHT TURN RIGHT oh nevermind GO STRAIGHT"
What do you mean by "nagivation issue"? I don't see the navigation system changing momentarily and then back (that would be analogous to your example). If in your phrasing the navigation system includes object recognition then if it istructs the car to suddenly steer right without any real reason, how could we trust that it stops before hitting pedestrians? Even if in this situation it would've corrected itself, I wouldn't say that this is a minor issue.
Right, nothing should ever be built unless it can be 100% correct immediately.
I’m stunned that it hasn’t, so far as we know, actually hit anything yet. I’m not sure how, but clearly they’re doing something right between choosing drivers and writing software.
There were several high publicity collisions with a Tesla colliding into stationary objects -- a highway barricade [1], a blue truck [2], a parked police car[3], two instances of a parked ambulance[4]...
Teslas driving themselves hit objects all the time.
So, devil's advocate. We know the Autopilot ignores stationary objects, so a lane with a parked vehicle (emergency or otherwise) is therefore a reasonable place to travel at the set speed on the cruise control, etc.
But, I believe this discussion is about the new FSD software which is supposed to be more capable. Have we had reports about the new one doing the old tricks?
"autopilot" versus "full self driving* capable" versus "full self driving" versus "autonomous mode" seems like marketing hype instead of actual improvements. After all, "autopilot" was supposed to drive itself, so what's the new one do differently?
Interestingly, the original creator of the video has since made it private. FSD beta testers are carefully handpicked — a lot of the Tesla “influencers” are given access for some free marketing. I wonder how many FSD bad driving videos are not uploaded to YouTube at all because they don’t want to say anything negative about Tesla and possibly lose influence/ad revenue.
This is on top of Tesla classifying FSD as level 2 system (while promoting as “level 5 soon”) so they don’t have to share data with CA DMV. Only reason not to be transparent like the others is if you’re not confident in your system.
There's a mirror of the original video here: https://streamable.com/grihhc. I consider it fairly valuable to study academically, as it shows both how the Tesla autopilot system performs, and how the the driver reacts to the system, with the knowledge that it's a beta, and their manner of movement, operation, and knowledge of the system and what they let it do.
I think the biggest problem I have is that the driver doesn't intervene/even really stop the car. The good autopilot 'test channels' like 'AI DRIVR' will routinely disengage if any safety concern is present.
The monorail test driver, in contrast, knows the monorail is a problem and was criticized for not correcting the car driving under the monorail, which is already illegal behavior.
Do you know if the terms of those arrangements give Tesla copyright ownership of the videos? Just wondering whether the driver or Tesla issued the DMCA.
The driver is a big fan of Tesla (even in that video he was very impressed with FSD 10). I think he probably deleted it himself because of the backlash
This is near the infamous monorail where previous versions of FSD would kamikaze into the giant concrete pillars. Presumably in response to the monorail failure Tesla updated their obstacle avoidance to handle unclassified objects.
From the nav you can see the car is trying to go straight but swerves around some invisible obstacle. I wouldn’t be surprised if this was a failure in their new “voxel based” collision avoidance.
It seems likely that navigation is momentarily seeing the straight-ahead as obstructed and trying to route around using the street on the right? But the car really needs to respond more calmly when doing navigation changes (see especially the car in the OP video - no safe driver changes course in a split second in the middle of an intersection, not at that speed).
Voxels? Wow. I was recently contemplating conscious driving vs ones own autopilot (when you get to work and dont remember any of the drive because it was unevetful). I realised that I sometimes drive largely in image space and other times in world space. It was an odd thing to deliberately flip between them. I don't really want to think about that...
I think the whole dream/goal of building a self driving car on top of a system completely designed for manual human interpretation and input is asinine, and that's before you even consider the complexity of adding other humans manually driving or suddenly walking in front of your car from a blind spot. Instead of building a digital system from the ground up, we are forcing manufacturers to come up with tech to read speed signs meant for human eyes and accurately interpret them under a myriad of conditions. What if it's covered in mud, or snow? Wouldn't coming up with national/international standards and a digital system work better for digital cars, like broadcasting speed limits instead of only relying on interpreting visual inputs, etc? Or standardizing the charging interface so you don't end up in a mac-like dongle hell and can just know no matter what car charging station you pull up to it will just work, like gas stations and standardized fueling systems work for all brand of ICE vehicles? I can go to any branded gas station and there isn't a station specific for just Ford cars. It just seems like a mish mash system of cludges hobbled together instead of a solid, well thought out system from the ground up. Due to that, we are making it 10x harder (and more dangerous) than it really needs to be. It's like saying you have to build all modern OS's on top of DOS made for 8 bit cpu's.
That's the entire problem. These systems don't understand anything about the world. If you see a sign covered in snow you're going to recognize it as such and make your best guess as to what it says and weather you care. You probably don't need that information to drive safely because you have a lot of knowledge to fall back on. These things dont, so some folks want to improve GPS accuracy and maps and signage. That's not a viable solution on so many levels.
General AI is the only way forward for fully autonomous cars.
> If you see a sign covered in snow you're going to recognize it as such and make your best guess as to what it says and weather you care.
The first winter after I got my license, I was driving around at night on a road I hadn't been to before.. kind of in the middle of nowhere, unpaved unlit back roads.
I saw a sign, partially covered in snow, and had a hard time making sense of it. Why this odd shaped sign in the middle of nowhere? I really wanted to figure it out and fixated on it a bit too much.. only right after making it past the sign, I realized it's warning about a tight bend in the road right ahead. Almost made it into the ditch. Thanks, sign, I would have seen the curve just fine if I had just focused on the road ahead.
Right now, the TSLAs with FSD will disable it in snow+rain, so I assume that even if TSLA achieves L4, it will be only under ideal conditions until they have enough data to resolve these edge cases.
I keep seeing people incorrectly call less than perfect driving conditions "edge cases." That term refers to rare conditions. Rain, snow, pedestrians, and support pillars are not rare.
You're right, they are not really edge cases, but the original poster mentioned a sign covered in snow. In that case, a human would probably just do the same thing the TSLA would: blow the stop sign or continue at the current speed limit until they get pulled over.
It seems like currently the minute it detects rain it shuts it down. (wipers are automatic also)
On the other hand, there are a lot of videos of FSD Beta on youtube that show the car driving into the sun where it's almost impossible for a human to see, but the cameras detect all of it.
Agree with your points but one thing to note, electric car world has indeed settled on a universal fast charging standard. Its also just as ubiquitous, thankfully.
Every new EV from Rivian to GM to BMW uses the “Combined Charging System.” Tesla, chooses to continue to use their own proprietary design in the U.S. despite having CCS technology available. (and the implementation completed already - European Tesla models ship CCS - required by law.)
Tesla will sell you $10K beta software that wants to steer into pedestrians, that most choose to pay for because after delivery, the price doubles (so act fast!) Most who fork that kind of cash are not able to access all parts of this beta software, this software license cannot sold/transferred to the next owner regardless of what the maroney sticker says, but don’t worry, FSD is being launched next month.
If thats excusable, surely then everyone’s ok with vendor lock in for DC Fast charging unless one wants to shell out $450 for a slower “old school” Chademo adapter, the fact one cannot buy replacement parts for their car, and cannot repair their own vehicle.
When your technology starts working only when the entire world changes dramatically, you probably don't have a great technology idea.
In particular, road maintenance is already expensive and often left to rot. Still, current driving is relatively robust - even if a few signs are damaged or go missing, no major issues will occur. Road signs keep working even during storms, even if there is no power, even if something has hacked national electronic systems, even if the GPS is not accessible.
Replacing all this with brittle, finicky, expensive, easy to attack en masse digital systems is not a great idea.
Cars had value without the need for major changes in the world. Road signs were codified and universally implemented only after cars proved their worth. Roads were not invented for cars, road-ready cars had existed for a good few thousand years before the first actual car.
Yeah, everyone remembers the famous 8 lane interstates built by the Romans, cutting through Europe, with gas stations every 16 miles. The only thing we needed to adapt were the road signs!
So you see no qualitative difference between a couple of horse carriage crossing the North American plains, and an Interstate system (that requires no stopping) with gas stations, motels, parking lots, malls, suburbs?
Have you looked at maps of cities with roads and parking lots? Cities in north America are literally built / rebuilt around cars, cities in Europe heavily modified.
Or, conversely, that means you find any advance towards autonomy in cars, in their current form, such as lane assistants, or working well in good weather on interstates, etc. is completely worthless?
There is a huge difference between a technology that is useful in the current world, as cars were at their inception, but can be significantly be improved by infrastructure; and a technology that requires a huge change in the world's infrastructure before it becomes useful.
That is, if autonomous cars can improve safety or convenience on current infrastructure, they will prevail, and new infrastructure will develop with them in mind. But if they are only useful after such changes, they will likely linger for decades before any adoption.
You did not answer: Do you find any advance towards autonomy in cars, in their current form, such as lane assistants, or working well in good weather on interstates, etc. is completely worthless?
Autonomy today is in the same situation that cars once were; even the current state of 'autonomy' has obvious advantages already and will only become more in the future.
Edit: How do you think cars were useful at their inception? Slow, with a range of a few miles, needed to be rebuilt every few miles, gas to be bought in pharmacies. Quite a bit had to change before cars became something practically useful beyond showing 'I am rich'.
Tesla (and fanboys) say that they are clear about the limited capabilities of Autopilot and FSD when in reality, they indirectly support all fan-videos by letting them live on YouTube while DMCAing critical videos.
They want you to believe that FSD is more capable than it truly is.
I think most likely the person who created the video must have done it. I don't see how Tesla would own the copyright. This also isn't the first time one of these videos has gone viral, and the other ones were not taken down.
I have FSD but not the beta. When cars cut across my lane autopilot will not react for a solid second or so, then proceeds to slam on the brakes after the car is clear. It does not instill confidence to say the least. From my POV it feels like someone put a debounce on the reaction to prevent phantom breaking, to the point the car doesn’t brake when it needs to.
The part that always bothered me most about AP on the highway was how willing it is to drive full speed into an obvious problem. Obvious for me, that is. Dense merging traffic, for example -- I can see the conflict coming, I can read the way people are driving on that on-ramp, and I take defensive action. My Tesla would just drive right into the center of that mess than then panic when things got crazy.
I no longer have my Tesla, but I stopped using AP before I sold the car, partly as a result of this behavior. And partly because phantom braking became pretty common for a number of overpasses we go under on our way to grandma's house, and it really scares the crap out of my wife & kids.
Aside from phantom braking, the best use case for AP, in my opinion, is to be a jerk and cruise in the middle lane of the freeway. It deals okay with that most of the time.
well, you don't just "trust" it, if you are smart enough you quickly learn the good and bad sides of it. When I use it (95% of time) I know what to expect and I have a habit of putting my foot on the accelerator pedal instead of break pedal for "phantom" braking which has gotten a lot better than let say in 2018. You also know the limits - if the lines look strange - no lines at all or if they are not perfect or if there are extra painted lines then you have your hand on your knee with a finger holding the steering wheel ready to quickly grab the wheel and go. You also learn as a pro tip to let the car push its limits and let it do what it wants to do like doing something dangerous then realizing that it would correct itself after your "comfortable" zone. People expect a "human" like behaviour from FSD/AP but it is not, it is far from it. It is a good/ideal assistant but not a full replacement.
I wish tesla did some training for everyone on autopilot to show them what to expect and how to handle it.
> You also learn as a pro tip to let the car push its limits and let it do what it wants to do like doing something dangerous then realizing that it would correct itself after your "comfortable" zone.
That's literally "trusting it".
As a human, you are attempting to learn the situations under which the car operates safely. You could be very conservative and say "it never works". The big problem is that the kinds of errors that occur in these systems are very different from the kinds of errors that humans expect. Even designers of autonomous systems can be surprised by emergent behavior. You say you can quickly learn the good and bad sides of it, however, that's not true. How much reliability do we expect from a system that, when it fails, leads to sever injury or death? ASIL-D is something like one failure in 10^9 hours [1]. That's 100,000 years.
You do not have that much experience with your Tesla, sorry. In principle, you might drive every waking hour for 10 years, never witness an accident, and you still not have an argument that passes even the most basic scrutiny for safety for critical systems. It could make a fatal mistake in the 11th year and that would not be a big surprise.
No one has that much experience with the ASIL-D components of any car. That's why I we must rely on regulation and standardization to help keep us safe. Safety isn't just "well I tried it a bunch under these conditions and didn't get hurt". And that's why what Tesla's strategy is reckless.
If I have to pay more attention to the car while I'm attempting to trust it to adequately respond to a situation it put itself into, then why would I be using AP in the first place? It's not relaxing, it's making the trip more stressful.
Sometimes I wonder about the folks who think AP makes for a more relaxing drive. I think that's only true if the level of defensive driving it does matches well with the driver's innate driving style. I'm not prone to having close calls on the road because I avoid them well before they get exciting. I've ridden with people who just don't drive that way, though. They'd probably be okay with AP.
I worry a little bit about things like this. If they were just using debouncing on a band-aid rather than dealing directly with the bad input that led to the phantom breaking, then that would seem to imply that the project is falling into the classic death march project trap. It turns out that the last 10% of the project is going to be 90% of the work. Meaning that the original schedule was off by an order of magnitude. But the business has made made huge promises that mean that the team is not at liberty to adjust the schedule accordingly. And so they're under ever increasing pressure to cut corners and pile on quick fixes.
Production AP is a completely different software stack than FSD beta. I think the only things that carry over are the car detection, VRU, and traffic light CV NNs.
I also have FSD but not the beta and I just wanted to add that my car (a 2020 model X) does not do this. Actually I've always been very impressed with how it smoothly slows to avoid merging cars.
Is it possible you have older autopilot hardware that runs different/older software?
I have the latest hardware and software. Note I was not talking about merging cars. I’m talking about someone cutting across my lane from a perpendicular street, so they can enter a lane adjacent to mine.
It wasn't even close. Look at where the ped is at 12 seconds, compared
to the car. It's because he stopped the turn, and straightened the vehicle that the pedestrian stopped and looked up with concern (at 13 seconds). Even if the pedestrian broke out into a full run, they wouldn't have been in any real danger.
This sort of close interplay between pedestrians and cars is very common in cities.
The concern I have is that the "eye contact" element is lost. When in doubt of another motorists intentions, we make eye contact and look for acknowledgement or an indication of intent. This doesn't exist with FSD.
Edit: Some people have pointed out the car wasn't supposed to be turning right which I missed. If that's the case, the situation and my opinion is completely different.
The pedestrian stopped as soon as they saw the car suddenly veering towards them; not just once the driver intervened, as far as I can tell.
Whatever the hypothetical (it certainly might have missed the pedestrian, or made a last second emergency stop) - it suddenly veered towards a pedestrian that was already walking on the road, and that alone is absolutely not OK. Scaring the living daylights out of people isn't acceptable, even if you might not have killed them without an intervention. And let's be fair, if the pedestrian had not paid attention, and the driver neither, this certainly could have lead to an accident. Even if it were "just" an unexpected emergency stop; that itself isn't without risk.
People aren’t bothered by how close the car came to the pedestrians - it’s the fact that the autopilot was programmed to go in a straight line and at an intersection decided to steer into a crosswalk.
Not to mention the fact that it seems entirely legal for a company like Tesla to test such dangerous functionality on public streets with other drivers and pedestrians, with zero permission or regulation.
> the autopilot was programmed to go in a straight line and at an intersection decided to steer into a crosswalk.
Agree this is a weird and very concerning bug.
> Not to mention the fact that it seems entirely legal for a company like Tesla to test such dangerous functionality on public streets with other drivers and pedestrians, with zero permission or regulation.
This driver seems to be someone who I would be ok with testing this. He's got his hand an inch from the steering wheel and obviously paying attention. I would rather this scenario happen, than Tesla be restricted to no public road access and they just unleash an untested system.
The problem, as I see it, is that the longer driver uses the autopilot feature the less attention they will pay to it as long as it doesn't do something nuts like this. My understanding is that this occurred soon after a new 'beta' update to the autopilot system. I'm not sure how this is surfaced to the user, if you need to opt into this new version etc.
My fear is that a new version of the autopilot system could have new bugs introduced that could disrupt the navigation on a route that the user has confidence that the previous version of the autopilot could handle. They commute on route x every day and over time they've gained confidence in the autopilot on that route. This new update however is going to run them into some obstruction due to a bug. That obstruction might be those monorail pylons we see in the seattle video or it might be a crosswalk. A driver who had confidence in the autopilot might be well distracted having confidence in his automation and not be able to correct the situation before tragedy.
IMO autopilot should be banned until it can be proved to be 100% safe. I don't think we can get there until roads are outfitted with some kind of beaconing system that sensors in the car can read and cars on the road are potentially networked together cooperatively... and only then to be enabled on roads with those markers/beacons.
People in this thread deciding that the system is safe because it's no worse than a drunk driver or student driver are missing the point. We absorb those hazards into the system because they present only a handful agents in the system. Out of 100,000 drivers in a rush hour flow, how many are students and/or drunk.. probably very few. However as teslas keep selling with this feature, our new class of hazard agents keeps going up and up and up.
Hell we wouldn't even have to mandate it to be illegal at the political level. Perhaps the insurance industry will do it for us.
Just curious, how does one prove 100% safe? Even if Tesla's were show to be safe on every road in existence there are always too many variables. Weather, construction, road condition, other drivers, objects falling into the roadway, on and on.
There is some level of reasonable safety that should be expected, but proving 100% safe isn't a realistic goal, nor is it even a standard for existing automobiles.
I'm not sure you could get to 100% safety. See any rail system in the world, however I'm more comfortable with our general rate of derailments I think (I haven't looked into the statistics). So I think we could get to a 'safe' system that looks like 'automated freight carrying in a special lane that is restricted, has built in 'hinting' and only if all the freight carrying vehicles are networked' Let the truck drivers get in that lane, fully engage automatic pilot and hopefully one day take a nap, to be awoken upon approach to their destination where they will take over, move out of the special lane, exit the freeway and proceed to their dropoff/pickup.
Almost hitting is an exaggeration, but making a right turn while pedestrians have started crossing at an intersection is a 100% incorrect (and likely illegal) action. It's concerning to me that the "full self driving" car cannot successfully handle this very common everyday occurrence.
That's definitely illegal and definitely happens all the time.
Except it usually happens when the pedestrian has already passed the area where you'd turn through or the intersection is enormous and the pedestrian just started crossing the other side.
I'd expect FSD to follow the law in this case though.
Going behind pedestrians in a crosswalk is legal in many states. It's universal that you must yield to pedestrians in a crosswalk, but not universal that you must wait for them to clear the crosswalk.
Going behind pedestrians in a crosswalk is legal in many states.
Not in the state in which this particular Tesla was being driven:
"The operator of an approaching vehicle shall stop and remain stopped to allow a pedestrian, bicycle, or personal delivery device to cross the roadway..."
"when the pedestrian, bicycle, or personal delivery device is upon or within one lane of the half of the roadway upon which the vehicle is traveling or onto which it is turning"
It further defines "half" to essentially mean "all lanes going in the same direction as you" which makes it illegal to cross if the pedestrian is anywhere in the crosswalk on a one-way street.
Interestingly enough that should make it legal to pass in front of pedestrians who are crossing traffic going the opposite direction, provided that you do not interfere with their right-of-way.
It is illegal in seattle (and all of washington state), where this video was filmed to enter a crosswalk when there are pedestrians present in the crosswalk.
We don't know however if Tesla would have stopped in front of the crosswalk as AP was disengaged(Which was the correct thing to do)
We know that pedestrians were far enough not to feel threatened as they waved back responding to drivers wave. Police was at the intersection also and they did not look concerned either.
I agree, but this is probably the worst behavior I've seen of the FSD software.
It abruptly tried to do a right turn on a no-right intersection with no warning (map said it was supposed to go straight, it looked like it was going to do that right up to the last second).
Putting aside the "almost hitting a pedestrian", this is really dangerous behavior for an autonomous vehicle.
It's not like the car is going rogue and just randomly making a right turn. It's trying to find the right side of the road in a rare intersection. A monorail in the middle of a city street is quite rare so this is an understandable failure-mode.
Look at the screen. It is going rogue, the line on the screen shows where the car is supposed to go, it abruptly decided it was going to take a right turn.
I look at current driver assistance systems as human-robot hybrid. Any driver assistance system should be required to be clear in it's intentions to the driver and give time to react, otherwise it's like a wild horse.
I don't know where you get the idea. Even plane autopilot will not land the plane w/o Pilot's supervision.
People who don't read manuals put themselves at risk.
Even simple systems like Cruise Control carry plenty of warnings that one should be familiar with when operating (from Ford's manual):
WARNING: Always pay close attention to changing road conditions when using adaptive cruise control. The system does not replace attentive driving. Failing to pay attention to the road may result in a crash, serious injury or death.
WARNING: Adaptive cruise control may not detect stationary or slow moving vehicles below 6 mph (10 km/h).
WARNING: Do not use adaptive cruise control on winding roads, in heavy traffic or when the road surface is slippery. This could result in loss of vehicle control, serious injury or death.
[and few more, shortened for brevity]
I don't think so. Most people are aware that autopilot exists for airplanes, and they are also aware that there are always human pilots on board the plane.
It doesn't really matter what you think. Studies have shown that a significant number of people believe that ADAS branded as "Autopilot" means that the driver does not need to pay attention.
I'm also not sure where you get that idea, it's clear when purchasing that Autopilot is a fancy cruise control to assist drivers and that full attention is required at all times.
Why did the car want to swerve right in the first place? You can see the route map on the display, it was going straight through this intersection.
EDIT - other people are pointing out the "no right turn" sign but I believe that's for cars to the left of the elevated tracks. The sign hanging by the traffic lights indicates the crossing road is one-way to the right, so that turn should be legal from where the Tesla is.
But it isn't planning a turn here so I don't think you can say "it would've made a tight turn and cleared the pedestrians." It looks like it has no idea where it's going and could have wanted to make a wide turn into the left lane, or even be swerving far out of its lane and then back in because it interpreted the intersection as something like a lane shift.
In this case pedestrians waved back to the driver who apologized as they were far away. If they were closer car would have probably stopped as AP stops and slows down near pedestrians.
But what does "yield" mean? It means slow down, and proceed when it's safe to do so. If a pedestrian starts crossing a 4-lane road and I have time to turn across his path without making him stop or deviate, then I can proceed safely and I think I've met the definition of "yield"
In Washington, where the video was recorded, the rule is that you must stop and remain stopped as long as there are pedestrians in the crosswalk in in your half of the roadway or within one lane of your half of the roadway.
"Half of the roadway" is defined as all traffic lanes carrying traffic in one direction. In the case of a one-way road such as the one in the video "half of the roadway" is the whole road.
In your 4-lane hypothetical, if there are two lanes in each direction you can drive through the crosswalk if the pedestrian is in the farthest away opposite direction lane from the lane you are in. In a 4-lane road with 1 lane in one direction and 3 in the other, you can drive through if you are in that 1 and the pedestrian is in the farthest 2 away from you. If you are in one of the 3 going the other way, you have to stop no matter which lane they are in, because they are either in your half (the 3 lanes going your direction), or within 1 lane of your half.
It is interesting how different the law is from behavior. Walking through Seattle, I would not expect that this was the law considering how often people will cross in front of and behind you (both of which I’m fine with).
It's true that there are laws and enforcement, but an automated system needs to obey the laws. Anything else is a judgement and machines shouldn't be making judgements when lives are on the line.
In the state I live in it is illegal to enter a crosswalk when there is a pedestrian in it when the road is undivided as in this example. I find it unlikely that there is any state where it's legal to do so for a one-way street such as this. (There are states that treat the crosswalk as two halves when there is two-way traffic.)
Ehhh at most that would be one of those things where, "yeah, that is the law but it's pretty impractical and unnecessary for people to follow to the letter and so everyone breaks it when safe to do so and it's a dick move to actually write tickets in those cases."
If a driver is trying to turn right, and the pedestrian has just entered the crosswalk other side of the street, 40 ft away, it seems stupid to wait for them to clear the whole 40 ft when your right turn doesn't put them at risk. I say that even when I'm the pedestrian in that situation.[1]
If the worst thing about SDCs that they do these kinds of violations (which would also include going 62 in a 60 mph zone), then I would say SDCs are a solved problem.. Though to be clear, Teslas are not at that point!
[1] Not just idle talk -- I cross East Riverside Drive in Austin a lot on foot, where this exact situation is common.
The concern I have is that the "eye contact" element is lost.
As a motorcyclist, runner, and cyclist, I can tell you that drivers will look you right in the eye as they pull out in front of you.
we make eye contact and look for acknowledgement or an indication of intent.
That behavior stands a good chance of getting one killed or hurt, especially in Seattle, the land of "oh, no, you go first..." Look at the wheels, not the eyes. The wheels don't lie.
As a seldom bicyclist I've seen this as well. But it's gotta be a lot less frequent to see a driver make eye contact and then proceed to almost kill you versus one that yields. Even as a seldom bicyclist (and as a driver even) if I'm approaching a four-way yield if I can't get eye contact with someone else approaching the intersection I'll just yield to them by default.
The car is going way too fast towards the crossing. No safe driver pretends to drive straight and then suddenly tries turn and sneak in front of pedestrians on a crossing. A person driving that way is a jerk :)
Agreed it wasn't close, but it's still not the kind of behavior you'd want to see in a robot car. If I were a passenger in a car where a human driver did this, I would wonder if they were impaired, and I would no longer trust their driving abilities. Tesla's whole selling point is that a) autonomous cars are safer than human drivers b) we are almost there. This clip shows both claims are questionable at the moment.
I wasn't close because the driver was paying absurdly close attention. We can't say if it would have been close or not, but there's certainly no hint the car had a handle on the situation, and it's clear to see it was making some kind of mistake, even if possible a non-fatal one.
Good thing evidentiary videos of car accidents are copyrightable. We wouldn't want to discourage auto-collision-footage-creators from creating content by failing to protect their exclusive monetization rights.
The car probably thought the pedestrian was an emergency vehicle, given the person was wearing a bright red coat and Teslas on FSD have a habit of crashing into them.
To Downvoters: Well it is actually true. [0] and just recently another crash involving an emergency vehicle. [1]
So there is a strange habit with Tesla FSD and red objects in its view. Given those incidents, care to explain why I am wrong?
Because even in cases where cars have hit emergency vehicles, it's because the software didn't see the vehicles at all and just continued driving straight in its lane. Whatever flaws the Tesla vision system may have, the idea that it is programmed to deliberately seek out and crash into emergency vehicles seems pretty far-fetched (much less that it would mistake a person wearing a red coat for an emergency vehicle and therefore attempt to crash into it); I assume this is why people are downvoting you.
The no right turn sign appears to be for the other side of the street, since at the intersection itself, there's a one way sign indicating right turns are possible, and no do not enter signs.
Some of the discourse suggested that previously it had had problems with the pillars for the monorail running down the median, and the owner/driver was trying it again to see if it had improved.
One of the big limits of this kind of AI is that it does not provide human-legible explanations for its actions. You cannot put it in front of a tribunal.
Actually, all kinds of data pertaining to the decision making process is recorded (at least for some of Tesla's competitors, not sure about Tesla), and in great detail. The data is specifically designed to make the AI driver "debuggable", i.e. it includes all kinds of details, intermediate representations, etc that an engineer would need to improve a poor decision, and thus certainly to understand a poor decision.
Whether that kind of logging is always on or was specifically on here, I don't know, but I'd expect Tesla can analyze why this happened: the car does have the ability to explain itself; it's just that owners and drivers do not have access to that explanation.
The box isn't as black as you might think; they're not training some monolithic AI model, there are separated systems involved. Also, the models aren't entirely freeform; i.e. engineers embed knowledge of how the world is structured into those networks.
They can use those intermediates to project a kind of thought process others can look at - and you've probably seen videos and images of that kind of thing too; i.e. a rendered version of the 3d intermediate world it's perceived, enhanced with labels, enhanced with motion vectors, cut into objects, classified by type of surface, perhaps even including projections of likely future intent of the various actors, etc.
Sure, you can't fully understand how each individual perceptron contributes to the whole, but you can understand why the car suddenly veered right, what it's planned route was, what it thought other traffic participants were about to do, which obstacles it saw, whether it noticed the pedestrians, which traffic rules it was aware of, whether it noticed the traffic lights (and which ones) and how much time it thought remained etc.
...at least, sometimes; I don't know anybody working at Tesla specifically.
And while they emphasize their lidar tech, I bet Tesla's team, while using different sensors, also has somewhat similarly complex - and inspectable - intermediate representations.
IIRC, in the incident where the Tesla [Edit: Uber self driving car] collided with a pedestrian pushing a bicycle in Arizona, the Tesla repeatedly switched between calling the input a pedestrian and a bicycle. And took no evasive actions while it was trying to decide.
>the incident where the Tesla collided with a pedestrian pushing a bicycle in Arizona
That was Uber's self driving car program. Notably, the SUV they were using has had pedestrian detecting auto-stopping for several years, though I'm sure it's not 100%
>That sign applies only to the lanes to the left of the pillars. It is legal to turn right there from the right lane. I've done it myself. Yes, it is confusing.
That sign applies only to the lanes to the left of the pillars. It is legal to turn right there from the right lane. I've done it myself. Yes, it is confusing.
The correct term is ‘Fools Self Driving’ given its users think that they are driving a ‘fully self driving’ car yet they have to be behind the wheel and pay attention to the road at all times.
Not exactly the FSD robot-taxi experience the Tesla fans were promised. Instead, they got happily mislead with beta software.
From the comments in this thread the Twitter copy was not the original creator of the video. The original creator posted it on YouTube but has since made the video private. And they DMCAd the person that reposted it on Twitter.
I imagine they probably started getting a lot of negative attention.
Anyone is physically capable of sending a DMCA takedown.
However it is supposed to be from the copyright owner [17 USC 512(c)(3)(A)(i)] and if it isn't, it can be ignored [17 USC 512(c)(3)(B)] and makes the person who does it liable for damages [17 USC 512(f)(1)]. Although that's probably pretty hard to use in practice.
Your YouTube support link isn't about the DMCA, and in fact is an attempt at trying to get people to use their system instead of the DMCA.
YouTube's internal copyright takedown system is just that: Internal. DMCA is a legal avenue that you can use against YouTube and their users, but it wouldn't be via their internal copyright take down system.
It is in YouTube's best interests for you NOT to use the DMCA as there are avenues where it can leave YouTube themselves civilly liable (OCILLA[0]), so they work hard to funnel people into their internal copyright system that doesn't expose YT.
OP’s point is that anyone can lie in the DMCA takedown form and say they own the content. It’s illegal but I’m not aware of it ever coming back to the perpetrator in the form of jail time/lawsuit.
Is there any evidence for the video being DMCA'd (and if so, who sent the takedown - IIRC youtube shows that), instead of being taken private by the uploader as the link in the tweet indicates?
The tweet that went viral was a clip of a video posted by a Youtube channel owned by someone else. Seems pretty obvious the copyright owner who would make a claim against the tweet was the Youtube channel who owned the video.
This looks like a weird navigation failure. It certainly shouldn't have made this maneuver at all, but it's provably false to say it would have run over the pedestrians... this frame shows it:
Context - this is after the sudden sharp turn, as the driver is taking over.
- We can see the system is still driving itself; the driver has not taken over and is not a factor. The blue steering wheel shows this.
- The dotted line coming off the car shows its intended path. If you step through the video, you can see it decide it's going to make the turn (for some unknown reason).
- Before the driver takes over, as this frame shows, we see the car decide to turn back to the left and resume its correct course. In fact, if you step through the video, you can see the car is turning the wheel to the left, rotating under the driver's hands before the driver applies enough force to disengage.
- The car was aware of the pedestrian at the time it decided to make the turn (you can see them on the screen), and its clear the path would not have collided with the person even if the driver had not taken over, and the car had not corrected itself.
- But the car did correct itself, and intended to resume its straight course before the driver took over, as that frame above shows.
This is not "Tesla almost hits pedestrian!", this is "Tesla makes bad navigation decision, driver takes over before car would have safely corrected"
> but it's provably false to say it would have run over the pedestrians...
You can't prove something like that, I mean consider the irony that you typed this statement right after
> It certainly shouldn't have made this maneuver at all
How can you prove that it wouldn't have made another unexpected maneuver?
It's all speculation, and that's OK. I wish OP used a different title, because everyone seems to be getting hung up on it, when the lede is certainly the FSD being unable to maintain a straight line.
I neither see nor hear any indication of a turn signal, so I do not think it is actually trying to make a right turn.
Besides, the pedestrians were already on the road when the car initiated the turn. Note the "one way" sign: that street only has a single lane, and that lane was already occupied. It should definitely have yielded.
Agreed that I don’t think it was trying to turn. Crazy how the system just went for it even with pedestrians. However, I know this area well; that street it’s turning on to is actually 3 lanes with no parking on the right except for buses.
Erratic navigation and driving. 1 second earlier the car was heading straight. The car placement was firmly to the left edge of its lane. A safe driver would slow down towards the turn and use the right hand side of the road - pedestrians will recognize this "body language" of the car and understand the situation better.
In our peripheral vision, the car placement and "body language" is probably more important than ostensible signals such as the turn signal (!)
This kind of "body language" is something we learn when driving and automatic drivers should adhere to it too, so that they can be predictable and interpretable drivers.
I don’t know what where you live, but here in the civilised world you’re not legally supposed to turn on to a road when there are freaking pedestrians already crossing it.
You’re supposed to wait for them to fully complete their crossing, and yes: the overwhelming majority of drivers here abide by that requirement.
The car didn't cross the zebra path, so it still had time to stop and be compliant.
I drove in US, and I know how slow and relax the traffic is. Come drive in Brussels or any of the big European cities (like Paris) and see for yourself.
Good to know US is civilized and EU is not. The difference is that you just have way more space than us.
Basic game theory dictates that if automated driving is seen as too safe, pedestrians will feel comfortable walking anywhere, anytime. Dense downtowns will become completely gridlocked and unusable for cars. Which maybe isn’t a bad thing. But if you want cars in cities you need them to be perceived as dangerous. Yeah I was convinced by Gladwell.
I agree, but also think we will adapt in many different ways. New types of tickets. There would be no go zones where pedestrians don't compile(or comply!). Mackinac island will not have this issue! And for fun I have to note basic game theory also dictates if we replace airbags with a sharped steel spike that comes out during accidents people will drive safer!
I love reading about self-driving cars and cryptocurrencies on HN. Suddenly, all the orders-of-magnitude improvements in efficiency/performance/scaling/production/integrity/etc. go out the window. Apparently they get replaced with conversations that devolve into how bad humans have been at the tasks that the fanboys are hoping get replaced by marginal-at-best tech.
E.g.,
git discussion on HN: of course we all know how git leverages merkel trees to make it workable to produce millions-of-line ambitious projects across timezones with people we've never even met before (much less are willing to fool around giving repo permissions to). But goddammit it takes more than five minutes to learn the feature branch flow. That's unacceptable!
self-driving car discussion on HN: I've seen plenty of human psychopaths nearly mow down a pedestrian at an intersection so how is software doing the same thing any worse?
Based on the audio in the video it seems like they were testing a Tesla FSD update? It sounded like they deliberately drove it in an area that it did not handle well and even commented that it was 'better'.
It seems like it thinks the lane shifted to the right because it had to reason to do a right turn.
This isn't the standard autopilot but the "full self driving" option that is currently in a limited beta (around 1000 users that have gone through extra approval, not just 100% random owners)
iirc the car does all the driving but you have to have a hand "on the wheel" so they don't technically have to legally classify it as a self driving car or something.
I'm guessing if he is videoing it and hovering over the wheel that much this has happened before and he is nervously expecting it?
Tesla has caused a lot of confusion by calling this system "full self driving", but in reality, this system is an SAE level 2 driver assistance system where the human is required to be in charge of driving at all times.
In the video there is a steering wheel symbol near the speed that is highlighted in blue. That indicates autopilot is engaged, when the driver turns the wheel to correct the mistake the symbol turns gray indicated autopilot disengaged.
The guy in the video is stress testing the system by repeatedly driving next to the pillars which he knows are handled poorly (previous versions of FSD would actually drive right into them). So he is being extra cautious with his hands on the wheel.
This tech is clearly not ready for mass adoption. I am surprised, to say the least, that they are allowed to sell
cars with that, clearly not safe tech, on board.
Where are the regulators?Pretending that it is all good?
Does anyone know if there have been efforts to connect a city's traffic lights/data with a self driving system like this? That crosswalk had a timer on it, and if that status could be accessed by the Tesla in real time, the autopilot would have known not to make that turn even if the cameras were telling it something else. Idk if those things are networked, but it seems like a good investment to do that if it means safer self-driving cars.
Haha, I commented on this exact sort of video, one titled something with "success" where the car somehow managed to get past that monorail but in an extremely sketchy way, way worse than any learner driver would every do.
This FSD beta seems way too cocky and at the same time confused underneath, I think we will see quite a few incidents with this one unless everyones a good boy and is really in the loop with taking over quickly.
I think I've seen a similar video in a previous FSD version and the car tries to swerve into the concrete pillar, so I think this person was trying to stress test that same situation, which I guess you can say it improved. I'm assuming this situation had something to do with the pillar again. Who knows though the reroute was really weird.
This will have to be settled in court eventually (I'm guessing very soon, looking at all these videos). It's very unlikely that car manufacturers will be able to avoid all responsibility just because of fine print in the terms of service.
FSD without LIDAR/radar is a fool's errand. Camera-only systems just do not pull in enough data to drive on real roads around real hazards.
For one they do not actual approximate human eyes since our eyes themselves are gimbaled in our skulls and our skull is gimbaled on our necks. Our eyes can switch their focus very quickly and return to scanning very easily.
Not only are we processing the visual information but also feedback from all of our body positioning and other sensory inputs. Driving heavily relies on our proprioception.
Fully autonomous cars need a lot of different sensors and processing power. Even driver assist should be using a lot of sensors. Autonomous driving needs more and better sensors than humans because it needs to deal with humans that are both smarter and more irrational than the automation. Besides conscious abilities humans have a lot of inherent self-preservation instincts that might be locally optimum for survival but globally poor choices.
Tesla scrimping on sensors because their boss is an egomaniac is the height of hubris. A web service moving fast and breaking things and choosing cheap but redundant options is fine. Usually the worst that happens is people lose money or some data. Moving fast and breaking things in a car is literally life and death. Musk's hubris around FSD is sociopathic.
Even worse, we actually rely heavily on our knowledge of the world, common sense,and physical intuition to make sense of our vision.
Put a human in a room with no shadows, unrecognizable objects, and misleadingly sized objects (tiny cars, huge animals), and then watch them fail to judge distances or other vision tasks.
This is Fifth Ave and Lenora in Seattle (about a block away from Amazon Spheres). Lenora is a one-way and you can take a right off of 5th onto Lenora (basically heading toward the water). The signage (and road in general) is confusing because of the monorail pylons.
Totally naive guess, but could there be some misdetection of a road feature (Stoplight)? Car swerves very close to when people in the background wearing very bright neon and very bright red jackets cross each other.
Selfvdriving cars almost hitting a pedestrian is terrible. Socialmedia platforms is an attention economy, where the most spectacular posts get attention. I think its diverting attention to the wrong causes.
Wait, why did the steering wheel and the car swerve right here?
Because the pedestrian that was passed or because the one waiting to cross?
And wasn't the drivers intent to drive straight ahead?
On the video - "almost hitting" is a VERY misleading way to phrase it, I see worse driving on a daily basis from human beings. However, I'm glad for the high visibility and scrutiny because we should hold FSD to a higher standard and the pressure will create a better product. (I drive one and use self-driving)
I agree this was not even a close call, also maybe its all due to driver interrupting the autopilot, but this video still shows to many problems for FSD.
A self driving car almost hitting a pedestrian is terrible. Socialmedia platforms is an attention economy, where the most spectacular unusual posts gets most attention! I think its diverting attention to the wrong causes.
People are going to die. I’m okay with it. These people are not hand selected to be sacrifices. If single deaths can do stall progress the west is going to get nowhere.
Recently I have been wondering why there are no videos of FSD v10 on any of the mainstream platforms which apparently includes HN now. There are tons and tons of videos of FSD doing amazing, amazing things. The situations it handles are unbelievable. There are tons of videos of it making end to end trips with few or no interventions all while handling insane environments that would break every other self driving system. If you showed these videos to someone in 1990 they would exclaim that the car “drives itself,” regardless of the knowledge that a person has to supervise and that it makes mistakes. We have arrived. And there isn’t any sign of it on cnn, Reddit or hacker news. But what you do see on these platforms are the handful of cases where FSD made a serious mistake. The overall picture painted by these platforms is incorrect.
According to Tesla's own legal documents FSD beta is still at level 2.
Level 2 automated driving is defined as systems that provide steering and brake/acceleration support, as well as lane centering and adaptive cruise control.
So it is the same as in most modern cars (other brands also have 'environment awareness') but Tesla still market it as 'sit back and close your eyes' driving.
Almost hitting pedestrian is a big stretch. Car had enough time to stop before he took hold. He probably just picked up the wheel because he didn't want the car going that way.
It's barely a stretch. The car veers pretty significantly. The driver and the pedestrian both notice this and wave as a 'sorry'. The car may have stopped, of course we won't ever know, but it's absolutely wrong to say that it was just the wrong direction - again, both the driver and pedestrian acknowledge the aggressive turn.
Car wasn't at a very high speed and if breaks are working normally it would have a lot of time to break before hitting the person. Almost hitting a pedestrian is a big stretch of a title, we don't really know what the car was going to do. Probably there's some logs on the car that will tell Tesla if the car would stop or not, but for the viewer of the video we don't really know.
I think we’re missing the point that this is currently designed for a driver to monitor at all times, the driver intervened appropriately, and thereby provided another training example for the network. This is also a beta version that is being tested by humans who have the legal and moral responsibility for control of the car.
One of the attributes of human perception is that it is TERRIBLE at maintaining persistent vigilance without engagement.
Even at a very low level, the nervous system is designed to habituate to constant stimuli; e.g., when you first encounter something that smells (good or bad) it can be overwhelming, but after a few minutes the same smell barely registers. More on point, spend some time looking forward at speed, or rotating (e.g., in a swivel chair), then stop quickly, and watch how your visual system creates the illusion of everything flowing in the opposite direction.
Now, scale that up to higher levels of cognition. The more the car gets right, the worse will be the human's attention. When a car here does almost everything right, people can and will literally read or fall asleep at the wheel. Until that one critical failure.
As a former licensed road racing driver and champion, I find the idea of anything between L2 and L4 to be terrifying. I can and have handled many very tricky and emergency situations at a wide range of speeds on everything from dry roads to wet ice (on and off the track) — when my attention was fully focused on the road, the situation, my grip levels, the balance of the car, etc.
The idea of being largely unfocused while the car does almost everything, then having an alert and having to regain, in fractions of a second, full orientation to everything I need to know then take action, is terrifying. 60 mph is 88 feet per second. Even a quick reaction where I've squandered only a half second figuring out what to do is the difference between avoiding or stopping before an obstacle, and blowing ~50' past it, or over it, at speed.
Attempts to say "it's just fine because a human is in the loop (and ultimately responsible)" are just bullsh*t and evading responsibility, even if human beta testing is fantastic for gathering massive amounts of data to analyze.
Among high risk and speed sports, it is almost axiomatic for us to draw distinctions between "smart crazy" vs "dumb crazy", and everyone knows the difference without a definition. The best definition I heard was that it's the difference between [using knowledge, technology, and skill to make a hazardous activity reliably safe] vs [getting away with something]. You can 'get away' with Russian Roulette 5 out of six times, and you'll probably get a great adrenaline rush, but you can't expect to do so for long.
Although this kind of "full self driving" has much better odds vs Russian Roulette, it is still unreliable, and the system of expecting the human to always be able to detect, orient, and respond in time to the car's errors is systematically unsafe. You will 'get away with it' a lot, and there will even be times when the system catches things the humans won't.
But to place the entire "legal and moral responsibility" on the human to 100% reliably operate a system that is specifically designed against human capabilities is wrong, unless you want to say that this is a [no human should every operate under these conditions], like drunk driving, and outlaw the system and the action of operating it.
If your concerns are correct, shouldn’t we see a lot MORE collisions among the millions of current tesla drivers using the existing, less advanced system than we do among comparable vehicles and drivers? Wouldn’t we expect to see higher insurance premiums for tesla drivers with FSD than for comparable drivers and comparably expensive cars? That doesn’t seem to be the case for the most numerous tesla vehicles[1]. In which case this sounds like a “it works in practice, but does it work in theory?” kind of situation :)
Indeed - one thing insurers are good at is gathering good and relevant data! In this case, a quick skim shows the Tesla often more to insure than the regular car, but not a ton. What I'd want to see is the data for only the models with the "Full Self Drive" option.
Not necessarily more, but we do see some really horrifying ones that humans would rarely do. E.g., the car in FL that just full-self-drove at full speed straight under a semi-trailer turning across the road, decapitating the driver, or the Apple exec that died piling into the construction divider on the highway because the Tesla failed to understand the temporary marks on the road.
I'm fine with Tesla or other companies using and even testing automated driving systems on public roads (within reason). Ultimately, it should be better, and probably is already better than the average human.
My objection is ONLY to the idea that the human driver should be considered 100% morally & legally responsible for any action of the car.
Aside from the fact that the code is secret and proprietary, and even it's authors often cannot immediately see why the car took some action, the considerations of actual human performance make such responsibility a preposterous proposition.
The maker of the autonomous system, and its user, must share responsibility for the actions of the car. When there is a disaster, it will, and should come down to a case-by-case examination of the actual details of the incident. Did the driver ask the car/system to do something beyond it's capabilities, or was s/he unreasonably negligent? Or, did the system do something surprising and unexpected?
In the present situation, where the car started to make a sharp & surprising turn and almost ran over a pedestrian, it was Pure Dumb Luck that the driver was so attentive and caught it in time. If he'd been just a bit slower and the pedestrian was hit, I would place this blame 90%++ on Tesla, not the driver (given only the video). OTOH, there are many other cases where the driver tries to falsely blame Tesla.
We just can't A Priori declare one or the other always at fault.
Watch again. Those pedestrians were in the crosswalk before the car turns. That's illegal at best, and definitely dangerous for those pedestrians. The car wasn't taking a tight right turn either.
That's not what I saw. I saw the car abort it's turn that would have taken it through the crosswalk in front of the people and not impeding them. Still illegal, but a far cry from almost hitting them.
The car aborted (or was aborted by the driver) and stopped before finishing it's turn on the road from what I saw, leaving it at about a 45 degree angle instead of the 90 degrees it would eventually reach if it continued turning.
I think it's the devil in the details here. "Was aborted by the driver" is the key thing, nobody is arguing this error was so bad it could have killed civilians even if the driver was responsible.
Was it though? Look closely, and maybe replay it a few times (I just did, another response to a different comment of mine had made this point so I looked). The car seems to have recovered itself, and the driver grabbed it a second later after it was already correcting.
That it decided it needed to turn all of a sudden us perplexing, and something that should be looked into, but it also went back to a straight path a moment later, either because of pedestrians too close or because of some other reason.
> nobody is arguing this error was so bad it could have killed civilians even if the driver was responsible.
"drove directly through where the people were going to be" most definitely implies that to my eyes. People die for exactly that reason all the time. Maybe I'm confused as to what you're trying to communicate.
If you substitute those quoted words into the sentence I was responding to my meaning becomes less dramatic:
> The driver took over way before the car entered the crosswalk section.
+
> "Entered the crosswalk section" is a rather euphemistic way of describing "drove directly through where the people were going to be"
reply
=
"The driver took over way before the car drove directly through where the people were going to be."
If you are right that the car stopped itself then it's certainly not as large an error as I had assumed. I think what I was mainly trying to communicate was simply a disagreement with the person I was responding to who implied there was little wrong with this behaviour at all.
For one, before the car enters the intersection, there is a pedestrian crossing. There is a pedestrian on the right side about to cross. I think it is safe to say in most countries, the car has to yield and stop.
I suppose we'll never know if the Tesla was going to follow through on that right turn.
But if you were an observer on the corner and could not see whether it was a human or an AI was driving the car, you would almost certainly consider whatever the car did (starting the turn) while pedestrians were actively crossing dangerous.
"Car doesn't yield to pedestrians". For "almost hit" I would expect car to run by pedestrian in less than half a meter or for pedestrian having to react quickly to avoid car.
It makes more sense when you realize the car wasn't even supposed to be turning. Look at the nav screen. If this guy didn't take over, where exactly was the car going? I think it's very easy to speculate it wouldn't have ended well.
Yes, it's weird that car decided to turn right. But I don't agree that's it's easy to speculate that it would hit the pedestrians, they were pretty far. I would think some emergency braking would take over or something. We have many thousands of tesla cars out there, lots of people are making videos about autopilot, but I don't think we have an example of a car actually hitting a pedestrian?
It looks like it re-routed down the side street at exactly that moment. You can see the projected path line shift to that street immediately before.
It doesn't look like it would have hit anyone to me (it was only partway through turning, so was facing people when it or he aborted it, but that's not the path shown on the display), but it was definitely an illegal move.
Step through the video frame by frame; you can see the car begin turning back to the left before the driver takes over. He actually stopped the left turn; the car would have recovered quicker without his actions.
Not that the driver was wrong to do what he did; we have the advantage of frame-by-frame replay. But the frame-by-frame does show that this is not what it appears at first glance.
The dotted black line coming from the front of the car (which I am assuming is the intended route) quickly snaps from straight ahead, to a hair pin right, to a normal right turn.
Ignoring the fact that the right turn happened to be into a pedestrian crossing with people on it - what was the car even trying to do? The sat-nav shows it should have just continued forwards.
I am astounded that software capable of these outputs is allowed on the roads. When could crossing a junction then taking a hairpin right back across it, ever be the correct thing to do?