In my case it's my Forgejo instance. My blog is static files with a limited amount of pages to crawl and indeed it's no issue at all. Forgejo is a dynamic service with an infinite amount of pages to discover and it runs git in the background to generate (some of) the pages. Tons of bots can easily overwhelm my small server. I could hide the repositories, but they're open on purpose (it's open source after all).
I have it guarded by a simple cookie check now, which seems to work well enough. Only a small subset of bots pass it. It likely kills search engine discoverability, but that's the tradeoff I have to make.
Copying what I posted under the original[0] that no one noticed because it's quite relevant to your mention of UTC for past events:
The naming of "timestamp with time zone" is one of my favorite pet peeves. It's one of those things that you can say "well technically it's true" about.
The article suggests that for past events, UTC and this timestamptz would be acceptable as a general rule, but even there it depends on what you will be doing with the data. If you intend to interpret it as a series of local occurrences and try to visualize/summarize that data later, you may be in for a surprise as your user has moved to another timezone and now all the past events are translated to the wrong local hours [1]. For example, your system might end up showing that the user's best time for jogging based on historical data is at 2 in the night.
But if your user does jog from 2:00am to 3:30am, your localized data is going to look wrong in the fall in a timezone where the clock changes from daylight saving and the 2:00am hour repeats itself.
It’s easier to convert historical records from UTC because the official rules don’t change for past dates (there may certainly be data errors in tzdata … what’re gonna do … )
If my user does jog at 2 AM, then it's better to show 2 AM than 7 PM or 10 AM even if there is a slight chance that on one singular day 2 AM might repeat itself. The user who is aware of how daylight savings works will surely be able to figure that out.
Technically you might argue that I should show 1 AM if the user did run at 2 AM in summer time, but everyone I personally know keeps their schedule over DST transitions. That is, if they did something at X o'clock before the transition, they'll keep doing it at X o'clock after it (sleep be damned). So generally showing 2 AM would be the most correct solution.
You cannot get that information from just UTC if you don't know where the user was when they made those historical events. Thus you either have to keep a history of their location (complicated) or just store the local timestamps (or at least the offsets) at the time of event. Always being able to convert from UTC with no extra data assumes that the user will never move, which might be fine if your application is limited to users in a single country.
The naming of "timestamp with time zone" is one of my favorite pet peeves. It's one of those things that you can say "well technically it's true" about.
The article suggests that for past events, UTC and this timestamptz would be acceptable as a general rule, but even there it depends on what you will be doing with the data. If you intend to interpret it as a series of local occurrences and try to visualize/summarize that data later, you may be in for a surprise as your user has moved to another timezone and now all the past events are translated to the wrong local hours [0]. For example, your system might end up showing that the user's best time for jogging based on historical data is at 2 in the night.
Most of the “money American tech brings in” comes from the magnificent seven. US software engineering salaries are high even outside of those. In fact, it's high even in companies that are merely burning investor's money.
there are plenty of american cos that bring in tons of money that are outside the mag7.
vc funded companies pay high so they can grow and eventually bring in lots of money, and america has the deepest vc pockets so it reaps the rewards of the biggest exits
Why would they "normalize"? Do you think Microsoft, Amazon, Apple, Amazon, etc. are going to relocate to the EU or something? Are all the venture capitalists going to flock to Spain?
The mechanics driving compensation arent "normal." American pay is driven by the underlying mechanics. The USA didn't just randomly win at tech.
There are real factors that could reduce US compensation, but calling that "normalization" assumes the current gap exists for no reason. It exists because the US software industry is structurally different from most of Europe.
Globalization? Look at manufacturing, it moved to a country where things are a lot more affordable. In a world where remote collaboration gets easier and easier and you're able to pay software engineers half the world away a lot less there's no way it wouldn't have an effect on the domestic market.
feel like that narrative has died given the return of RTO. In person work is really valuable
and the talent is just better in the US on average (mostly because of immigration!), software is so levered one good Eng can 1000x the value of a bad one
If demand for software developers decreases due to AI, salaries are likely to decrease as well. Take the academic world for comparison, where supply of very smart people vastly exceeds the demand.
I suspect demand for software is nearly infinitely elastic, so far we’ve seen demand and comp for engineers increase as coding agents got better
Academia for comparison doesn’t make money…maybe a better comparison is HFT? Plenty of very very smart people playing a zero sum game, yet their comp has only increased
I already addressed this in my comment. There are real factors that could reduce US compensation, but calling that "normalization" assumes the current gap exists for no reason.
It's a game. You're supposed to click on the different route options that are presented to you on the bottom of the screen. That's why it goes slower the more options there are.
But I also just followed it for the first time from beginning to end, not doing anything. Because I assumed someone had already done the math.
While 7700 per hour sounds big, pretty much any dinky server can handle it. So I don't think it's a matter of DDoS. At this point it's just... odd behaviour.
especially for a txt file. I don't know anything really about webdev but I'm pretty sure serving up 7700 plaintext files with roughly 10 lines each an hour isn't that demanding
Forgejo does set "cache-control: private, max-age=21600", which is considerably more than one second, but I grant it uses the "private" keyword for no reason here.
For some reason, Facebook has been requesting my Forgejo instance's robots.txt in a loop for the past few days, currently at a speed of 7700 requests per hour. The resource usage is negligible, but I'm wondering why it's happening in the first place and how many other robot files they're also requesting repeatedly. Perhaps someone at Meta broke a loop condition.
As facebookexternalhit is listed in the robots.txt, it does look like it's optimistically rechecking in the hope it's no longer disallowed. That rate of request is obscene though, and falls firmly into the category of Bad Bot.
Note that the API is split into XSS-safe and XSS-unsafe calls. The XSS-safe calls [0] have this noted for each of them (emphasis mine):
> Then drop any elements and attributes that are not allowed by the sanitizer configuration, and any that are considered XSS-unsafe (even if allowed by the configuration)
The XSS-unsafe functions are all named "unsafe". Although considering web programmers, maybe they should have been named "UnsafeDoNotUseOrYouWillBeFired".
I have it guarded by a simple cookie check now, which seems to work well enough. Only a small subset of bots pass it. It likely kills search engine discoverability, but that's the tradeoff I have to make.