"One of our EFnet operators located a server that a client was using, the server was most likely hacked and/or used for illegal activities." -- so they contacted the ISP and the ISP sent their request to whoever owns the box.
That sounds entirely reasonable to me. Hetzner has no idea whether or not the box owner is the suspected hacker (even EFnet assumed the server was hacked) in the same way that EFnet has no idea about whether an Hetzner employee was the suspected hacker. So three possible outcomes:
(a) the box was hacked, so sending said info to the box owner was not a bad idea,
(b) the box wasn't hacked, it was the box owner himself doing bad things, in which case Hetzner just gave the box owner personal information about his victim, or
(c) the box wasn't hacked, and it's a Hetzner employee doing bad things, in which case EFnet just gave the employee personal information about the victim.
The operator passed on personal information without knowing if a Hetzner employee might be the bad guy. Hetzner passed on personal information without knowing the server owner might be the bad guy.
If the operator had contacted the server owner directly to let him know his box was hacked, the result would have been the same.
Why did the operator have the expectation that an uplink of the server wouldn't just forward the mail to the server owner?
Because Hetzner is an award winning business who sells professional hosting solutions. You'd have a point if the company in question was some dodgy garage operation, but Hetzner isn't.
If EFNet are going to report the issue then there has to be a trust worthy contact at some point along the chain of network ownership, and Hetzner should have been that point. Quite frankly, it's irresponsible for a company of that size to behave in that way.
No, Hetzner is a bottom of the bucket discount dedicated server company. They are the German version of ThePlanet, and will likely collapse under their own stupidity soon enough as well.
I've personally tracked botnets infecting thousands of machines back to command and control servers at Hetzner. Guess what happens when you send off the report? They forward it to the bot herder.
What would Efnet lose by redacting the name of the operator themselves before doing it? They still show timestamped logs, and [operator redacted]@ef.net would be enough to note that an operator reported the issue.
If the log included chat between the malicious user and the operator, the malicious user __already knows__ the operator's nick, so there's no additional information. If the log doesn't include that, then what reason is there to include the nickname of the operator?
Rather than expecting Someone Else to sanitize our information (which fields do you feel are critical to redact? real name? phone number? IP? zodiac sign? Made-up-nickname?), it seems like it's a good idea to proactively redact the information ourself before making that abuse report.
The operator is the one that sent the email. Who are you suggesting should have redacted his name before transmitting it? Magic fairies in the SMTP servers?
They're not an ISP providing a dumb pipe. They're a hosting solution providing the servers that are generating the attacks. The attack is originating from them, not personal equipment relayed over their pipes.
Unfortunately, the past few days many EFnet servers (and more are following) have had to ban an entire ISP, which has not happened in over a decade, if not longer. Naturally, something extreme must happen for this to be even considered. [... and the extreme thing is ...] the email also contained sensitive information about who this operator was, including nicknames
This is ridiculous. Why would an IRC op want to keep his/her nickname from becoming public? On IRC, even IP addresses are public. Hetzners reaction seems entirely reasonable to me, especially if the server in question was "mots likely hacked".
> Why would an IRC op want to keep his/her nickname from becoming public?
That isn't what they said.
> On IRC, even IP addresses are public.
This may still be true on efnet. It has ceased to be true on many IRC networks. And even where it is, VPNs, shell accounts, tor, and similar methods are readily available to make the IP address worthless.
> Hetzners reaction seems entirely reasonable to me
Please tell us where you work so we can all be certain never to report any sort of abuse to your employer non-anonymously.
How not? The forum seems to be down now but the announcement heavily implied that all that was "leaked" was the nick name of an operator, which they claimed could somehow be used to deduce the real name and address.
The problem is not the public knowledge of a nick, but the information that the person using that nick made the complaint. It paints a target on a person whose nick is very often publicly associated with a real identity.
Exactly. If efnet has problems with giving the names of their "operators" they should send such requests as "securitymanager@efnet" or whatever. Efnet can become the target (it is supposedly target anyway) but no real names in efnet must be "leaked" by email forwarding. If you don't "have the time" or common sense to protect yourself why do you expect more from others?
What do you think happens when an IRC op gets his name passed to that kiddie? Many IRCd gets hit with 50Gbit/s all the time because of kiddie targeting operators that make such decisions.
I have reported spammers/phishers to Hetzner in the past and experienced the same: Hetzner's default policy is to forward a complaint to the server's owner - which also kind of surprised and annoyed me.
On the phone, support staff told me that Hetzner sees itself just as an uninvolved messenger between both parties.
Apparently, their support ticket system automatically forwards all complaints to the server's owner without any way to opt out.
The support person offered two alternatives: send anonymous complaints through a freemail service, or send the complaint to the personal address of a support team member, so that they can manually enter it into the system.
Yes, this is pretty annoying.
As a Hetzner customer I think this is more than fair. If someone reports abuse from my server, hetzner should forward me the report. They do not have access to the server, so why in the world would they do anything but forwarding it to the person or company that does in fact have access?
Protip: Don't put your credit card number in the abuse report. Just put the logs or whatever is required to prove that there was in fact abuse from that server.
The problem is that if you have someone using Hetzner's network for spamming or other illegal activities, you'd expect the ISP to want to keep their network clean of these sorts of things and investigate. Instead, they just say 'whatever, not our problem'.
On top of that, though, forwarding the complaint to the server owner just makes a mess of things, because now the server owner can target you and make your life hell. They can send you threats, DDoS your mail system, flood your inbox, etc., all the while knowing that Hetzner has their back.
It's like if you saw someone assaulting someone in an alley and called the police, and the police took the information you gave them and gave it to the person who assaulted them. Now they know who you are, and they can come after you too. Now your only real solution is to just not report it when someone does bad things, because you know it's not going to get fixed and it's just likely to cause problems for you down the road.
> Unfortunately, according to trusted sources (ex-employees) of Hetzner.de, this is policy and not an exception. They have realized they can save money (by limiting attacks) by redirecting the attacks back at the person reporting them. That way, the hacker/cracker/kiddie using their services will not cancel their contract with Hetzner, and in return Hetzner will remain protected.
I don't think that this has to do with Hetzner needing criminal business but rather with Hetzner not wanting to shut down an entire server if a part of it has been breached.
Forwarding that complaint in it's entirety is definitely not best practice however making such allegations is neither.
That was roughly my take away. Hetzner looks really bad - but the explanation that they are incompetent and lazy sounds more likely than the criminal explanation.
And EFnet looks wronged. But also looks bad for spreading allegations against Hetzner of profiteering from criminals without much more substantial backing.
Sticking to the facts of the situation would have produced a more powerful, and damning condemnation.
This isn't court, I think it's reasonable to forgive humanity entering into the issue. Besides, this appears to be a citizenship issue, so banishment also comes as no surprise.
That announcement should have clarified whether the communication they sent to Hetzner explicitly requested confidentiality. If not, the incident is as much Efnets's fuckup as Hetzner's. Why did it even need to contain such sensitive personal information about the sysop in the first place?
Even if their initial email requested it, wouldn't you assume that a large provider like Hetzner has at least partially automated their abuse handling? I wouldn't count on a human actually reading it before it gets forwarded to the customer.
"This has worked very well due to our personal involvement with a lot of said organizations. If we find an abuser on IRC, we try to not only ban him or her, but also to contact the provider so that the problem is handled at the right end, often with the involvement of law enforcement, as was the case with Kevin Mitnick, t0rn and a lot of other well publicized hackers/crackers."
Of course it is. If someone cost you time and money and severely harassed users of a service you're offering by illegally attacking your servers, then I bet you'd be all for reporting them as well.
The problem with script kiddies is that they're too lazy to create anything of their own and thus cannot emphasis with how hard it is to set up and maintain a free public service. So they don't have any second thought about destroying other peoples hard work. Plus the pseudo-anonymity of being online and the ability to not look your victims in the eye make such crimes easy for even some of the more ethical kids to shrug off.
The only way to have an open internet where everyone is free to create and contribute is to rat out those who seek to destroy it. People like the aforementioned might enjoy all the perks of an open internet, but their actions have the opposite effect.
So I'm not only in favour of reporting abusers, but have actually reported attacks on my own servers to their respective ISPs.
I'm not an expert, but isn't it more accurate to say that freenodes and other networks have "patched" the flaw via chanserv and nickserv bots to mitigate the problems with the network model? I know efnet doesn't want formalized chan/nick ownership, but they clearly dont like being the world's #1 ddos sink. Why not just include a master network quorum function and a grace period under which reattached networks don't get to merge their mode changes.
Really? Hmm. It has been 5+ years for me, my bad. But what's the point of the ddos's then? I see netflow graphs of huge attacks on major efnet servers on a regular basis.
I'm honestly not sure why people still DDOS servers these days. Maybe just fun and games, it is EFnet after all.
Not too long ago there was an ircd exploit that killed servers and split 100% of efnet, which had never happened before. Not to mention its down 50% of its user base from a few years ago. Tough times for efnet.
Kiddies are DDoS'ing to annoy other users and to prove that they are capable of taking down the nodes.
All servers on the largest IRC networks are very well connected to the internet and servers that are not, wont be able to last long on these networks.
Some IRC servers wont allow you to create new IRC channels during a network split, so an attack where the network is being split constantly can cause quite a lot of annoyance for the rest of the users.
This seems like a ridiculous overreaction to me. Not only do I not see much wrong with an ISP forwarding abuse e-mails to the admin of the hacked server (who is probably a victim too), but I also find a bit hard to believe that the nickname of an operator is enough to "derive a home address".
as an operator of a large IRC network I am forced to keep my real name off my employer's websites, websites like github and my home address out of the public record (director's addresses and such like)
Maybe the op is German. In some jurisdictions it's required to give your postal address if you publish content publicly. Germany is an example, the fine is up to 50000 EUR, but the real risk is that any lawyer can send you as cease and desist in the name of consumer protection, typically this will be about 500 EUR.
That leads us back to the pseudonym discussions: Not every name that isn't on ones state-issued id is anonymous. With mine, even without clicking on my profile, you can easily find out my name and yes, my address as well.
If I connect to an IRC network (mostly freenode, or gnome/gimp) I'll be online as darklajid as well. I ran a tiny irc network a couple years ago and used the same handle. For me that's not at all hard to believe.
I can totally understand Hetzner for just forwarding abuse complaints to the client (for root servers, the hoster usually has no "emergency ssh keys"), so the faster the original owner of the server can boot out the hacker, the better. At least it's better than disconnecting the customer from the internet entirely, especially as "haxx0ring" a server is damn easy these days, given the numbers of aged Wordpress installs alone. Also, a server owner who knows his server can support me as a hacking victim better than the hoster support who often knows nothing about configuration details, OS, disk encryption etc. on the server.
But I also understand EFNet, that their emails got blindly forwarded is bad, too...
> I can totally understand Hetzner for just forwarding abuse complaints to the client (for root servers, the hoster usually has no "emergency ssh keys"), so the faster the original owner of the server can boot out the hacker, the better.
Removing confidential information from such an email should take ~2min, I think that’s time well-spent, especially given that email per se is not a real-time medium.
I guess their automated forward system regex-greps the IP out of the complaint email and then forwards the mail to the applicable customer(s), so this system actually is realtime.
Also, it places the burden of dealing with spam on the customer, so that the hoster doesn't have to do spam-filtering.
"[...] someone at Hetzner chose to forward
this complaint to the actual abuser him/herself. [...]
Unfortunately, according to trusted sources
(ex-employees) of Hetzner.de, this is policy and not an
exception."
I really hope this is not true.
In 2011 there started to appear pornography when I searched for my name in image search. The reason was that a stackoverflow.com scraper showed my answers (which I post under my real name) but my profile picture replaced with porn. I have no idea why he replaced the profile pictures in the first place, but anyway.
The scraper site was hosted at Hetzner. I phoned Hetzner. They told me to write to abuse@hetzner.de, which I did (in German). I received an auto-reply but nothing else happened. I tried to reach their phone support (09831 610061) several times during their business hours (Mo-Fr 7:30-18:00 Uhr) but no one picked up. I wrote several other mails to Hetzner but never received another reply.
After about seven months the scraper went down but I have no idea if it had something to do with my complaints because I never received any feedback from Hetzner.
I have know Idea who was behind the site. They could have found out about me anyway because I posted this publicly on meta.stackoverflow [1], but still: The thought that they might have learned about my identity through Hetzner is discomforting.
I was just musing that the only word in this headline recognizable to someone outside this small subset of the tech world is "all", and yet it makes perfect sense if you know all the words.
kline (K-line) basically [K]ills a network or IP block, forbidding them from connecting to the IRC server. (I think the term stems from the config files for the original ircd, circa 1988)
Just to clarify: most servers haven't K:lined the blocks, but D:lined them (even though the efnet.org article says 'klines'). K:line = kill line, D:line = deny line.
The difference is where the check happens, after the client has had a chance to identify and a identd response is received or not, or at the TCP level on remote IP address alone (D:Line).
D:Lines are less resource intensive obviously. Not much difference for the client though :)
It is an IRC term for a "kill line". If your host tuple (nickname, username and hostname) somehow matches the kill line, your connection to the server will be closed with a message on why it was closed.
Doesn't explain why efnet themselves are not at fault for failing to redact the details if they are so concerned. What if the attacker was Hetzner staff?
Should we perhaps have sent the request through anonymized pidgeon? When you send an e-mail to someone you can find out quite a lot of information from headers, from a nick or a given name.
When contacting ISPs some people attach contact information so that the ISP can request more information, maybe even make a call to verify identity and discuss the matter to help resolve it.
This doesn't mean that the information is intended for the abuser himself. I'd expect the ISP to send anonymized information/questions to the owner of said service, and not just forward "everything" (especially without discussion with the reporting party).
Reporting DDoS and abusers risks putting a huge target on yourself, your company, your servers, etc.
I specified redacting the details, i.e. the irc nickname.
If the information within the headers is that sensitive, the headers should probably not be forwarded from efnet to hetzner.
At its most basic their servers are attacking efnets and so should be considered hostile.
By all means pass on contact information to resolve amicably - but don't leave sensitive information.
There's no reason to pass on irc nicks or sensitive email addresses - what use are they to hetzner?
I have the same expectation of efnet as you have for the ISP.
If the issue is that efnet is being identified as the reporter to the rogue attacker, then that is not what is stated in the efnet admin's text.
EFnet didn't pass anything on. EFnet reported abuse, and the EFnet reporting staff members information was the information that was leaked/forwarded to the abuser. Not a third party to whom EFnet acted proxy.
Are there any actual standards for how abuse reports should be handled by service providers? What Hetzner is doing here appears to be pretty reasonable to me (it matches what companies like Level 3 and nLayer do).
I can understand why you might not want sensitive email forwarded to the abuser, but why would you send that information in your initial complaint? For all you know, the person you're reporting to is the abuser.
I got an abuse mail from Hetzner once (they mistyped my ip).
The original complaint was something autodetected by their own system, mailed to themselves. The original complaint was attached, header and all. Something about malware on an IP similar to mine.
Also, when will this be in effect, the server i tested from had no problems connecting.
EFnet has always been a network that promotes freedom of speech. One of the core pillars of a free virtual society is trust. Trust not only amongst ourselves internally, but an undying trust in the companies that allow their users to connect to our wonderful network. We have survived over two decades, in a world that is increasingly image- and video-based. IRC can offer neither of those. IRC is based on ideas. Ideas that are exchanged in text. With text, as opposed to images and videos, one has to be put extra effort into the subliminal, the meaning, the message. This has been our catcher in the rye, and we intend to protect this content-based communication form, for as long as it is appreciated by the hundreds of thousands who every day turn to IRC for philosophical debates, dating and just about anything you can think of (I’m sure a lot of the things in that last category does not belong here in this text, but you get the picture!).
We rely solely at the goodwill of others, as is the case with most things worth saving. There is no money to be made. We all do this for free. Sure, some companies might have benefited from a small level of advertisement, attracting customers to their products. But all in all, it has mostly been an uphill battle against enormous attacks, sometimes exceeding 75Gbps of DDoS. This has made it impossible for all but the largest organizations to host a server on our network, or any other large virtual society. We are Don Quijote and the weather mills are often winning.
One of our key strategies is to preserve a close relationship with the major Internet- and Hosting Service Providers, as those are the networks that our users connect through. This has worked very well due to our personal involvement with a lot of said organizations. If we find an abuser on IRC, we try to not only ban him or her, but also to contact the provider so that the problem is handled at the right end, often with the involvement of law enforcement, as was the case with Kevin Mitnick, t0rn and a lot of other well publicized hackers/crackers.
Unfortunately, the past few days many EFnet servers (and more are following) have had to ban an entire ISP, which has not happened in over a decade, if not longer. Naturally, something extreme must happen for this to be even considered. Almost always can we find a solution through the use of good old fashioned communication. Alas, not in this case. Well, here is the story (to the best of my knowledge):
One of our EFnet operators located a server that a client was using, the server was most likely hacked and/or used for illegal activities. As IRC is often a playground for these people to use, before moving on to more serious targets (where they can make money through extortion), we take this extremely seriously. Because of the serious nature of this, our operator sent an email to Hetzner.de, a German hosting provider, to help them lower the abuse of their servers, as well as ours. This is usually a fruitful symbiotic relationships, where both parties stand to gain.
However, the big difference between this case and all the other thousands of cases we have handled in the past, is that someone at Hetzner chose to forward this complaint to the actual abuser him/herself. This might seem fair enough, as anyone accused should be granted the right to defend him- or herself. However, the email also contained sensitive information about who this operator was, including nicknames (from which names can be derived, and thus, also, home addresses). We know what an impact this can have on your social, not to mention your professional life. We have seen people lose their jobs, after constant attacks and we have also seen companies lose money that is hard to fathom, considering this is still just a simple chat for friends. This is a fundamental breach of that mutual trust that has allowed us to accept clients from Hetzner to use our network - free of charge, just like we do with anyone else wanting to connect.
This a give and take network, where mutual trust is vital for our survival. We are maintained by the community, and we exist solely for the community. Hetzner.de has broken one of the most fundamental aspects of any report of criminal activity or suspicion thereof; source protection.
I expect us to get attacked now, which will result in a lot of work for the company kind enough to donate money and time to continue to provide us with servers, in an era where almost everything else would be more profitable. But this is an ideological problem, more than a financial one. We have been attacked before, and we will again. We are prepared. But these preparations rely on the fact that we know who the enemy is. Hetzner.de has made that impossible.
As a result of this, we have decided to ban all Hetzner IP ranges (both IPv4 and IPv6) from our servers. It seems other networks are following, and I know QuakeNet has published a similar statement. We simply do not want anything to do with a company that values money over source protection and integrity. Some may argue that this was a one time mistake, and that we should not jump to conclusions so fast. Could this have been a mistake? Sure. Does it matter, given the consequences this could have had for this operator’s personal life and health? No. We do not appreciate cowards that would rather see someone else hurt, than take their responsibility.
Unfortunately, according to trusted sources (ex-employees) of Hetzner.de, this is policy and not an exception. They have realized they can save money (by limiting attacks) by redirecting the attacks back at the person reporting them. That way, the hacker/cracker/kiddie using their services will not cancel their contract with Hetzner, and in return Hetzner will remain protected. Left are those of us that work for free, and who will continue to do so, for as long as there are honest, reliable companies out there, willing to go the extra mile to protect the freedom of the Internet, and, above all, freedom of press and source protection.
Questions on this matter must be directed to Hetzner.de, as our involvement in this situation is over. This has been their decision based on questionable methods. It is unfortunate for them that they got caught, but it is good for the sake of the free Internet.
Sincerely yours,
Johan Boger, on behalf of EFnet and anyone else believing in integrity, source protection and a free Internet.
> the email also contained sensitive information about who this operator was
> ...
> This is a fundamental breach of that mutual trust
> ...
>Hetzner.de has broken one of the most fundamental aspects of any report of criminal activity or suspicion thereof; source protection.
Excuse me? How is this in any way Hetzner's problem? The only "fundamental breach" of trust I can see here is the one that occurred when efnet gave "sensitive information" on its own operators to a third party at all. It sounds to me like efnet is the one in need of a lesson on "source protection." This reads more like a "we screwed up bad and are about to divert blame as hard as we can" letter than anything else. But I'm more than ready to listen to any further context that can be given to this.
For abuse reports to be taken seriously you need to provide timestamped logs and legitimate real world contact details, in exchange there is the expectation that reports are not forwarded in an unsanitizied fashion to end users. (I worked an abuse desk for a few years)
The problem is that if you have a problem and want to solve that problem, the first thing you do is talking with the person.
Instead of trying to fix this like normal persons, you want others to work for you by free.
Who gives you the role of a judge in this case, to even consider that the actions from that ip where illegal.
If they were illegal why don't sue? If you ask a lawer the first thing he will tell you will be to talk with the other parties before starting legal actions.
I really hate to use this analogy, but consider abuse desks the police of the Internet for a second.
Your local ops team notices a lot of SSH break-in attempts coming from Sprint's network (for example). They forward the logs on to the abuse team, who reach out to the abuse team at Sprint. In the same way that a cross jurisdiction investigation would be handled by two police departments working together, the two abuse teams will work together to investigate and remediate the problem.
Now imagine if the Seattle Police Department contacted the Chicago PD about an ongoing investigation of a jewelry heist, and the Chicago PD says "we are too busy to deal with this" and forwarded it on to the thieves so they could respond directly and help the Seattle PD with their investigation... Now the investigating officer is compromised and at risk. Sure, he was a known as a cop to the bad guys before, but now his enforcement actions have made him a direct target.
This is a system that has worked for 20+ years. We have our own mailing lists, meetings and conferences, heck even a private direct dial phone system.
Except that the communication between police departments is clearly marked or known (because of various regulations/laws) to be confidential. Here it's not clear (as I've also mentioned in a previous comment [1]).
That sounds entirely reasonable to me. Hetzner has no idea whether or not the box owner is the suspected hacker (even EFnet assumed the server was hacked) in the same way that EFnet has no idea about whether an Hetzner employee was the suspected hacker. So three possible outcomes: (a) the box was hacked, so sending said info to the box owner was not a bad idea, (b) the box wasn't hacked, it was the box owner himself doing bad things, in which case Hetzner just gave the box owner personal information about his victim, or (c) the box wasn't hacked, and it's a Hetzner employee doing bad things, in which case EFnet just gave the employee personal information about the victim.
So, exactly why are they blaming Hetzner?