At high volumes, at my job, we have yet to find a good third-party log SaaS that performs not only better than self-managed Elastic but actually to perform good enough to be used.
New Relic could not handle the query aspect of having at around 5TB+ terabytes of logs (I know, a ludicrous amount of logs, but that's what it is) per day. Their architecture does not really allow for that. For small volumes I guess it would be enough. Not to ingest, that it did fine, but to query them under a reasonable time frame without a timeout, that's where it couldn't handle the high volume.
Also, their support service, while trying to win us over, that is, in their best moment, was nothing really stellar. Favouring sending us sales/presales people to solve technical problems.
Not advocating for this decision, but did you investigate Splunk? In my experience, that’s the paid logging service that competes with ELK. It will be expensive, so you have to consider the total cost of ownership (e.g., ELK requires some experienced people to run it at your volumes) but it works AFAIK.
I love splunk. Our clusters process 10s of billions of structured log events daily. We have search, reports, PagerDuty integration, dashboards, etc. It is crazy expensive but is the best system I've used in this space. We are having to save costs with so much data, so we are lowering retention time and moving the data to snowflake for data older than a week. More and more, we are leveraging Looker for reporting out of Snowflake and relying more on Prometheus monitoring for alerting. But Splunk would still be my ideal service if we had less total data.
This got me curious, so OK, Splunk's pricing pages are very obtuse and they are really pushy about getting you to contact sales directly to get bleeded, but I managed to get to this "actually has a number in it" page for their Log Observer services[0], and... it looks cheaper than NewRelic, especially at scale?
NR charges $0.25 per ingested GB after the first 100 free GB; Splunk apparently only charges a flat $0.10, if you choose ingest pricing.
I guess that NR includes (a free tier of) a bunch of alerts, monitoring etc. features in their package, while they're separate packages for Splunk. Still, that doesn't seem wildly expensive at a glance. Where's the catch?
I think for slightly lesser volumes Sumologic can be a choice. The kind of search queries, regex, and capture options give the feel of log parsing on a *nix box.
I remember having the same experience during my third job, in a french consultancy in Spain. For the sake of vindication (so many years after the fact xDD) I will just name it: Sopra [0].
So, for each business task (like adding some feature to the application we were developing), "Analysts" would write the "specification" in uml-like models in Eclipse (some kind of modelling plugin for Eclipse, I think it was a framework developed by the company over the EMF plugin suite), and a very short description of what this was supposed to accomplish, business-wise.
Our[1] task would be, then, to generate a scaffolding from these models (via Eclipse) and then to fill in the blank of the methods generated this way.
The problem lay at how the incentives were laid in that company (and many others like it, the whole IT "consultancy"[2] business, in fact) : Programming was a job that was poorly paid (but still, much higher than the median salary in Madrid, which is why I was there). So, every person had every incentive to get to either management or higher "design" roles (Technical Analyst, Business Analyst, Architect) as soon as possible. This meant that the people who remained in the company[3] were either the people that had 1 year, 2 year tops, of programming experience that got to now design whole systems and applications, or the sociopaths that floated to management by stabbing everyone on the way up.
You can imagine the quality of the systems produced this way...
Models were hilariously under specified, and clearly the person doing them did not have experience on how programas behave in real life. But because they were so insecure in their positions (remember the backstabbing and the lack of experience), any back and forth between the programmers and the analysts was handled with hostility and contempt from the analyst side.
Obviously we also looked at analysts with contempt. Still, they were paid better than us, so of course they were right and they had management support, and were told by our team leads to make it work anyway, so it was common to implement something completely different under the scaffolding, to avoid having discussions with the analysts, that checked that the scaffolding itself did not deviate from their designs.
So, my take in this kind of 5th generation (as they were called back then) code generation frameworks is that it encourages the kind of behaviour I saw. And when the same people that do the design also do the programming (the mason architects coders of our time [4]), this way of working is just redundant. Diagrams and models have their place in documenting a project, but certainly not in trying (and failing) to specify everything that can happen in a system.
Wow...I've almost written a blog post xD
-----
[0]: What a shitty place to work in, at every level. No joy was possible in that environment. Sociopaths thrived and naturally floated to the management positions. As we say in Spain: shit always float to the top.
I have so many stories to tell of that place...and I was only 6 months there!
[1]: lowly paid and lowly valued programmer-monkeys, literally the term they used when they thought we couldn't hear it (and for a more colourful spanish variation of it: "picatas" which was a derogative term meaning Typist)...how the tables have turned since then...
[2]: As we all know, it's just a sham. We were never consultancies, just the equivalent of cheap sweatshops for our french overlords, who came to Spain because programmer's salaries and labor rights were lower than in France. Still...they offered better salaries than the native "consultancies", so there's that. A common theme in Spain, actually. At least foreign exploiters treat labor better than our own national exploiters.
The situation has greatly improved since those times, and you can find genuinely good (foreign) companies to work at, in Madrid or Barcelona. The consultancies still exist, but they're no longer 100% of the job market now.
[3]: Because, naturally, the churn was stupidly high at the "picatas" level, who could find better offers and situations just by changing jobs every 6 months, which I did. I remember at that time how they tried to FUD us into "loyalty" by saying that they did not recruit people that had this kind of job-hopping CVs. It was false, of course, they were desperate to find people, so they took what they could. My father said the same to me. I don't doubt that was how it was at his time. That era of employee and employer loyalty had long passed by that time, if it ever really existed from the employer side in the first place (I doubt it, IMO it's just that employees had even less choice and were educated by society into loyalty for your task masters)
[4]: I'm very glad of how the industry has evolved, job-wise. These times are much more interesting from a technical point of view, than what existed in the past. I get enjoyment out of my job now, and that is priceless. Also the high salaries help. Not as high as outside Spain, of course, but high enough for a good life.
Thank you very much for this summary! I have a lot of polish colleagues in my work, and I always feel bad for not knowing how to start pronouncing their names aside from memorization (which I will promptly forget). This will definitively help me.
> and I always feel bad for not knowing how to start pronouncing their names
No worries, give it your best approximation and if it's way off or if someone cares a bit too much they will correct you, use the corrected version since then and you're golden.
We know it may be hard, no problem whatsoever and no one cares, don't worry about it one bit and definitely don't feel bad about it. I think poles actually prefer if you say their name in 'english way' rather than super-correctly in polish because it sounds more natural instead of being suddenly jerked back into polish for a millisecond and then back into english. Just remember the 'w'->'v' and you will get a lot of street cred for free
Oh, yes, this! I feel like I'm going crazy that my team members cannot see how incredibly slow Homebrew is. I don't know the reason for this, but updating the list of packages is excruciatingly slow and it does it every time I want to install anything.
Every time I have to use it, I have time to ponder about my life choices (like why am I using a mac to do sysadmin/dev work?).
I worked some years with Perl, and while it was fun most of the time, it had a lot of frustrating things that made it difficult to have consistent builds and dependency management. The "There's more than one way to do it" mantra is very damaging when all you want is a productive language to produce something, vs having fun implementing or tinkering with libraries in CPAN. There's a lot of research, including taking a look at the code to really know what the library does, to assemble programs in Perl.
Once you get over the initial conceptual barrier and remember how the sigils are used, and get used to the bless object system, it is fun to develop in it, on an intellectual level. But rather annoying on an "enterprise" level.
In the end, I find it very difficult to justify using Perl over almost any other language. Even PHP would be better for dev teams, due to the higher focus on productivity frameworks, even if the language holds no distinctive advantage over any other. I felt Perl is not really a professional language, but the product of a whimsical creator and community (which, by the way, is very welcoming and open).
While I feel sorry for the people that put the energy to make this happen and also for the people that may lose their jobs this month over this, as a native inhabitant of Barcelona, I can only celebrate the news, even if it is for this terrible reason.
When MWC is ocurring in Barcelona, public transport (subway) goes on strike (because of the maximum impact). But usually this only fucks over non MWC attending people, because tickets are so expensive that only people which can use a taxi anyway will go, greatly diminishing the impact of the strike for attendants while making inhabitants lives miserable for that time period.
Also, Barcelona already has a very severe problem of houses being rented out to tourists with very high markups, producing a gentrification of the city (along with other factors, not only the tourism is responsible for this). Events like MWC do not help this situation.
BCN is already overloaded with tourism, events like MWC only make it worse for those of us that live there.
I'm completely aware that this is not the responsibility of MWC organization or tourists, this is a regulation problem. But since regulation is stacked against us, I can only get happier that this year we won't have to suffer all the inconvenience of the event while reaping almost none of the benefit.
Thanks for the local perspective! I get the feeling MWC is a bit of a scapegoat here, tough.
> When MWC is ocurring in Barcelona, public transport (subway) goes on strike (because of the maximum impact).
So won't they just choose any other date with the next highest impact for their strike now? Why is that better?
> BCN is already overloaded with tourism, events like MWC only make it worse for those of us that live there.
100.000 visitors for 4 days, that must be less than 1% of Barcelona's hotel nights/year. It's also happening during off-season, so shouldn't lead to more housing capacity being taken away from the locals.
I get when people complain about the Olympics in their city or other costly events with unclear rewards, but this seems pretty harmless?
You never really know what you've got till it's gone. A hundred thousand visitors eating, drinking, commuting, sightseeing, dwelling, and flying about in your city in a short period of time.....for many cities that would be a welcome occurrence (apart from the traffic woes).
It's about $500m Euros of revenue, and I saw an estimate of something like 15k part-time jobs due to MWC. No small thing.
I'm also in BCN and everyone always says that but I'm not sure I buy it.
I mean, yeah, some guys benefit from having the MWC for sure but I'm not sure most residents do - I'm not letting out flats or running venues etc. I'm trying to get to work on time and pay my rent.
It reminds me of the similar arguments people make about the big cruise ships that come here - I mean sure, it massively benefits tourist attractions and restaurants etc. but most citizens don't see any of that money, they just get lungfuls of air pollution.
Isn't the sales tax in Barcelona 21%? Seems like it would be a big influx to the city's coffers that could theoretically provide it the means to do something like pay subway workers more and alleviate the need to strike in the first place.
If the money's not being spent in the right place, that's a local political issue that can't solved by anyone except the locals (e.g. those who can vote).
Honestly, the MWC doesn't bother me that much - yeah the metro always goes on strike but that's a problem with the metro unions and the Ayuntamiento not the MWC in my opinion.
But take the case of the cruise ships which do irritate me - every citizen gets to suffer worse air quality so a few can enrich themselves on the tourist money.
I understand that in theory that money might 'trickle down' to the citizenry at large, but it is far from obvious that this actually occurs sufficiently to compensate the downsides.
That money doesn't come for free. It's perfectly rational for people who don't directly benefit from a lot of visitors--but are inconvenienced by them--to wish to forgo both the visitors and the money they spend/taxes they bring in.
I suppose white collar workers who don’t rent out on airbnb don’t see any of that revenue. You could make an argument that they benefit indirectly when their local economy gets a boost and you’d probably be right.
But most people will look at the misery of temporary overcrowding, look at their bank account and think “no, I don’t want this” without considering second order effects.
It is very trendy for Barcelona residents (I am one also) to moan about the MWC or ‘gentrification’.
It is less trendy to complain about the relatively low unemployment and well-maintained civic infrastructure; both of which are boosted by a strong international business presence which employs a lot of people and brings a lot of wealth to the city.
Losing that international presence risks turning the city back to the dirty, poor, backwater it was in the 80s. If living with a couple of major international events a year is the alternative, I know which one I’d choose.
I think the cancellation of the MWC is a disaster for the city, not just in terms of short term economic loss, but in terms of longer term strategy, as the MWC is already muttering about following other events to Madrid or elsewhere.
I find it unfair to call it moaning as is if its a childish reaction. Most of my friends can no longer afford to live in the city itself with their non-IT jobs. The gentrification process is a real thing happening right now with very real consequences.
I guess that MWC is not the problem per se. And it is part of the reason of the exploding international presence at least in the IT sector. But it is part of the problem of the worsening in the quality of life for many people.
My argument is that much of the wealth generated by tourism doesn't necessarily benefit citizens of Barcelona. I'm sure a lot of people are getting a lot of benefits, but I'm not sure it is the people that need it the most.
Where does that attitude towards tourism come from?
I know this is politics but other places would welcome tourism and professional conferences with open arms.
When I see a group of tourists cueing in front of something in Copenhagen - sure - I might get annoyed but those people are bringing in money. Not to me personally. But that tourist money gets taxed or spent again. And ultimately it ends up being spent on a nicer park or causes some new cool restaurant to be viable, all benifical to me.
Eventhough we have extreme political parties here; noone is bad-mouthing tourism. Everyone knows we live of trade and we need to be an open and welcoming place; and it is out own problem to make sure our public transport can handle its peak traffic.
Really, I'm all for responsible tourism. Tourism is very good for the local economy when it is done properly.
But it must come with regulation to avoid encouraging foreign companies (or local, doesn't matter) buying all houses/apartments and then renting them mainly to tourists for a much higher price than locals can pay. Since Barcelona hosts a very high volume of tourists, they can afford to do this almost the whole year.
Please understand, I'm not complaining about tourists (although they come with their own problems, like a 24/7 unwelcome festive ambient in residential neighborhoods...), I'm complaining on how Barcelona is not able to absorb that massive amount of tourism and how it makes lives miserable for local residents, that are pushed out of the city because it is far more beneficial to land-owners to rent to tourists, and how restaurants/shops push prices up too, and how the center of the city is no longer targeted towards locals. And much of the city center's businesses are no longer locally-owned and bring outside supply chains and may employ non local people, so it is not even that beneficial to the community as a whole.
Yes, jobs are created, but tourism-related jobs are not famous for their ability for employees to be able to live in the city itself.
I know this is a local politics problems, one of electing people that will do the right thing, but it's not that easy unfortunately, since voting in spain is very polarized on ideological and tribal feelings. And meanwhile, locals will just suffer the negative aspect of all this while benefiting little.
From what I've read about Barcelona, it's more like living in a Disneyland, every day you're just swimming in crowds of tourists, from all over the world, bringing their idiosyncrasies. All (or a lot) of the shops are shops selling touristic trinkets. Want to eat at that nice restaurant? It's crowded, with loud Americans and Brits. 7 nights a week...
Part of it is that some of the people experiencing the downsides of tourism aren't getting adequate benefits from it. Or perhaps don't see how they're benefiting even though they do.
We're experienced with managing a big scale elastic cluster, but never managed the ML jobs, so knowing about this limitation is definitely useful!