EDIT: I didn't notice that this topic was about the DDG browser (which I didn't know existed) and responded assuming this was about the site/extension. For a browser, yes, a client-side solution is possible and probably preferable. Please check and upvote other comment trees.
This makes sense to me and is not alarming. Getting favicons actually is difficult to do robustly; many applications and websites use Google's service to do so, which then leaks the request to Google: https://www.google.com/s2/favicons?domain=ycombinator.com
Putting this logic in the client is not feasible. You want to send requests directly to every shady site that shows up in your search results, load their pages in the background, work through network delays and HTTP errors, and parse out the location/format of the favicon files?
DuckDuckGo hosting this functionality themselves is also a positive. They have previously been burned when the Web of Trust service they were originally using was found to be farming data, and turned it off immediately once discovered. Processing, hosting, and serving the icon themselves prevents that from happening again.
"This is not to say that DDG is perfect: links you click do seem to be redirected through a /l/ page on their domain ..."
I am surprised the user is not complaining about this instead of the favicons. Their privacy policy goes on about the privacy implications of Referer headers and instead of calling out browsers for sending Referer by default, they instead give themselves power to record all the user's clicked results themselves. The Referer problem is something that can be solved by the user at the browser level through, e.g., using a client that does not send Referer, browser extensions/plug-ins that can control headers sent, or perhaps with a local proxy to remove the Referer header.
Unless DDG has changed, these prefixed result URLs are the default. It is possible to get unprefixed result URLs using the "lite" version of DDG however that is not the default. "Privacy-focused" search engine chooses less private default. News at 11.
I recently noticed that DDG has started redirecting queries submitted via POST to /lite/. The redirect is to the same domain. No explanation. I have a custom client that does not follow redirects and I now have to submit two sets of HTTP headers instead of one.
These guys are trying to make money from advertising just like everyone else. They have to be very particular in the methods they use to do it -- check the exceptions in their privacy policy -- but it is the same game. Ads and affiliate links. That sort of business and privacy are always going to be at odds with each other.
This is done only for browsers that don't support setting the referrer policy. https://caniuse.com/referrer-policy. It should not be happening with any modern browser, and (like the other reply) I don't see it in mine (Firefox).
Are you saying DDG is forcing you to send a User-Agent header?
DDG's privacy policy also goes on about the privacy implications of User-Agent headers combined with IP addresses.
So let's say I take what they have put in their privacy policy to heart and I stop sending a User-Agent header. In response DDG sends prefixed result URLs? WTF?
Using haproxy, for example, I can use a "modern browser" and send no User-Agent header. I still get prefixed result URLs.
You can disable this on the settings page. Frankly I'd infinitely rather send my request through a redirect served by DDG than send my referer to every site I visit. I think this is a perfectly sensible default for the average user, who both doesn't block / obscure their user agent and doesn't have referer masking software in place.
You're free to disagree, but in that case you can make use of the option to change the default that they provide for you.
To get prefixed result URLs, use GET not POST. Try duckduckgo.com/lite/ and lite.duckduckgo.com/lite/
If I recall correctly there is a way to turn off the prefixed result URLs without having to turn on Javascript, by adding/changing a URL parameter, e.g., kh. However that is not the default setting. Not very friendly toward the privacy-conscious user who has Javascript disabled and does not read HN to find out about otherwise undocumented URL parameter usage.
For certain queries, I can actually get a different first result based on the HTTP method I use.
I'm not sure what you mean by bringing up HTTP method. The two main ways I know to get to the results page are to enter a query into the website or the browser search bar, and both of those seem to issue GET requests. They also both produce real URLs without any intervention. What defaults are different for you than they are for me?
I don't see how this is a privacy violation in any way. The headline is that DDG is tracking the domains you visit, but favicons are loaded for every search result you see unconditional on whether you click them, and DDG already knows what those results are when they serve them to you.
Which other browser doesn't fetch favicons locally?
And I have no clue how search results come into this? Of course the browser isn't fetching favicons on search result pages, that's the servers job, but that's not what this is about.
It's really odd that this is the top comment right now, given that even the headline makes it clear its about the browser, not the search results.
For the search results DDG can do whatever they way. As far as I understand from the title, this is for the pages that you actually visit. For those pages it totally make sense for the browser to figure out favicon locally on its own.
As for complexity, I'm sure users of privacy oriented software would prefer to not have favicon in small percentage of corner cases rather than leaking browsing history.
> Putting this logic in the client is not feasible.
> You want to send requests directly to every shady site that shows up in your search results, load their pages in the background, work through network delays and HTTP errors, and parse out the location/format of the favicon files?
Looking at DuckDuckGo search results and visiting a page that you navigate to are two different things.
1. DuckDuckGo search results:
DDG already returns the search results so there's no privacy violation to return the favicon or the URL for the favicon in the list.
2. Any page that isn't a DDG search results page:
Use client side logic to locate the favicon. This means worse performance but better privacy - which aligns with DDG's goals.
If you want to optimise this then DDG could send the client some precalculated Bloom filters with info about known sites. The client could use these to try certain methods of favicon retrieval first.
I don't need the favicon for a domain I never visited just because it adds some visual cool to your UI. That is not a thing. Cache it whenever I end up visiting the page and be done with it.
> Putting this logic in the client is not feasible. You want to send requests directly to every shady site that shows up in your search results, load their pages in the background, work through network delays and HTTP errors, and parse out the location/format of the favicon files?
I don't see why not. Browsers are constantly dealing with these very issues. It's one of their core competencies.
This makes sense to me and is not alarming. Getting favicons actually is difficult to do robustly; many applications and websites use Google's service to do so, which then leaks the request to Google: https://www.google.com/s2/favicons?domain=ycombinator.com
Putting this logic in the client is not feasible. You want to send requests directly to every shady site that shows up in your search results, load their pages in the background, work through network delays and HTTP errors, and parse out the location/format of the favicon files?
DuckDuckGo hosting this functionality themselves is also a positive. They have previously been burned when the Web of Trust service they were originally using was found to be farming data, and turned it off immediately once discovered. Processing, hosting, and serving the icon themselves prevents that from happening again.
This is not to say that DDG is perfect: links you click do seem to be redirected through a /l/ page on their domain, which can cause problems: https://lapcatsoftware.com/articles/duckduckgo.html