With Google Cloud, Serverless has been a reality from 2015! And welcome to Serverless to those coming from AWS.
Here is how Google Cloud achieves serverless.
Ingesting Data: PubSub. Scales to millions of messages instantaneously, no need to spin up and spin down the capacity/shards (like as in Kinesis), exposes RESTful interface for ingesting messages from anywhere (web/mobile ... ). If you want a high-performance interface, you get gRPC as well. With RESTful interface to consuming messages, you can connect PubSub to anything you want or trigger Cloud Functions / write to storage / ingest to Stackdriver (Monitoring system) using managed services.
Querying Data: Big Query. No need to spin up a single server. Just write your query in SQL and watch the magic of a thousand servers being spun up to serve the query in a fraction of second and process a petabyte of data in a couple of minutes. btw ... you can ingest data into Big Query in real-time and analyze results within in seconds.
Process Data using Dataflow: For workflows that are more complex than SQL queries to ones that need to be running continuously on streaming data. Dataflow is the serverless version of Spark. No more running our of memory errors, much fewer hassles with hotkeys, no more manual performance tuning of buffers, no more cleaning up log folders. If you are using Spark, but have not tried Dataflow, you are missing some serious magic.
Google Cloud ML for Machine Learning: Cloud ML is a hosted solution for running TensorFlow jobs. No more manual hyperparameter optimization, no more spinning up GPUs, no more scaling up the cluster size. It's all taken care for you.
Container Engine: Hosted version of Kubernetes. I am sure, everyone is aware of what K8 is and its capabilities.
I am from a Data / Analytics / ML background. The above was my reality of Serverless since 2015. Google Cloud has serverless options for Web (App Engine) / Mobile (Firebase), and other purposes as well.
Having done PhD in Cloud Computing and Big Data and I don't find Infrastructure/Big Data a sexy problem anymore. To a large extent, it's a solved problem. Building a data platform with a team of 3 people that can handle 10+ petabytes of data is easy. What is on the horizon and unsolved yet, is AI!
Would love to see if there are any other better/compelling Serverless options.
> Having done PhD in Cloud Computing and Big Data and I don't find Infrastructure/Big Data a sexy problem anymore. To a large extent, it's a solved problem.
This kind of sounds like you've never managed a production system, sorry.
There's also the whole "how do you process and manage lifecycle for an exabyte of classified/sensitive data" thing that nobody's nailed down yet. "Serverless" is not the panacea people make it out to be; as usual, it's marketing-speak for outsourcing ops, which isn't always feasible.
People who talk about "data platforms" in terms of size are like people who judge coder productivity in lines of code. I guarantee your 10pb "data platform" will choke and die the minute someone shows up with 10gb of historical data and you have to get the timezones right.
> This kind of sounds like you've never managed a production system, sorry.
Don't be sorry. Instead, challange me with specific problems you have and I will show you how to build solutions.
> how do you process and manage lifecycle for an exabyte of classified/sensitive data
The infrastructure for storing/processing data is in place. If you doubt me, please go ahead and give the above-mentioned tech stack a try and let me know if you still have any problems. Or give me a detailed description of what you have to do and I will tell you how to do it.
> People who talk about "data platforms" in terms of size are like people who judge coder productivity in lines of code
Let me be more clear. I know how to build the infrastructure for that scales of data. I am not saying that I will write the code for processing data. That depends on what kind of data you got and what you want to do with it.
> I guarantee your 10pb "data platform" will choke and die the minute someone shows up with 10gb of historical data and you have to get the timezones right.
> Building a data platform with a team of 3 people that can handle 10+ petabytes of data is easy.
I'd hate to see that server bill. I'm sure Google or Amazon would be happy to host that for you, at the cost of a full round of funding a month.
Point of reference: I worked for a company doing some basic ML analysis on the Twitter firehose; only a few tens of gigs a day. Their hosting costs were significantly over half of their overall operating costs (despite huge discounts, and including downtown New York offices), and they still hadn't gotten profitable when I left (they had actually gotten to the point where they were laying people off).
None of it was serverless either - it was measurably incapable of operating at the scale they required.
To me, the definition of Serverless is this: Developer writes code and the management of resources for running that code is taken care of by a Serverless platform. As a developer, I don't have to deal with servers, disks, networks, logs, metrics, ...
> As a developer, I don't have to deal with servers, disks, networks, logs, metrics, ...
Yes, until things don't work as they supposed to.
Disks and networks also don't matter if your application is simple. Disks are pointless if you don't keep a state. Networks will suddenly matter more when you start hiring its capacity.
If your application is simple, you can also run it on a single machine and don't care about any of the things you listed.
But just because those things were abstracted from you doesn't mean they don't exist and their limitations don't apply.
I emit logs. I mentioned that in my above comment as well.
> So your code assumes that the network/storage/DB are fault-free?
No, it does not assume that the network/storage/DB are fault-free, completely. It assumes they are fault free with the SLA limits provided by your Cloud provider. The software is written in a way that enables the platform to know about errors when then happen and remediate them. Like, when you build a website, you make the app serving layer stateless and choose a backend datastore that is replicated across regions and is highly available (like Cloud Datastore or Spanner). If your platform detects that a disk failed and your app is returning errors, then that instance is killed and an another instance is brought up. Very similar mechanisms exist for the above-mentioned data services as well for auto scaling, sharding, ...
If your Cloud provider can not guarantee the SLA's or shows a green sign even when the service is down, IMO they are not competent enough.
Awesome! Datastore connector, please!! Cloud Datastore is the ideal NoOps and Serverless storage solution for projects from small to large. It would be great if you can build a connector for it.
You can load *.info Datastore backups into BigQuery tables natively. You could probably write up a quick apps script to automate data refreshes it if you wanted.
Still not the same as reporting directly from a live datastore though :)
That's exactly my issue. I have a crawler that collects data and stores into Datastore. I need to periodically scoop up the snapshots and export them into BigQuery, then run the visualizations. I want my visualization to be live and also don't want the overhead of scooping the data.
Thanks! And yes, as you mentioned, the problem is that it works only for inserts. I use Datastore as my primary database. So requesting for supporting Datastore from Datastudio.
I guess that speaks about the scale at which they are operating in Google Cloud. Diane Green mentioned in a recent conference that one of their healthcare customers collect about 2 PB / user. Lot of companies struggle with managing / extracting value from data. Thats where the bottle neck is usually. If they have capability to handle more data, overtime their services evolve to collect, store and process more data. Once Big Data became reality, many companies started collecting orders of magnitude more data. With Google Cloud its easy to handle petabytes of data. That enables large scale computing companies on Google Cloud. (Think of driverless cars / genomics / large scale machine learning / social networks / ... )
That seems a bit too much storage, considering that Seagate only shipped around ~ 250 Exabytes worth of HDD in 2015[1]. Being extremely generous for world supply of Hard drive, we might have had only 1000 Exabytes of storage shipped in a year.
Assuming 2 Petabyte per user, mere 500,000 user will consume entire storage produced every year. May be they do, but 2 PB/user seems improbable. That's also $14k per person of data (assuming no discount from Google to this company).
Lets not assume that Google buys storage from Seagate. Google makes its own hardware for many things (networking, TensorFlow custom ASIC). Also I don't think that have that many number of customers.
Expert on Cloud Computing here. Short answer is "No". Here is why. Google Cloud has infrastructure that scales to petabytes of data and millions of users. Google primary uses this infrastructure for storing, processing and communicating the Internet. Add the services like Pub/Sub, Dataflow, BigQuery, TensorFlow & CloudML and things like security, communication backbone, ... its near impossible to build Google Cloud or even few critical components like the ones mentioned above with 2 billion dollars. Also, if they focus on building infrastructure, it might slow them down significantly. They are better off building their chat platform rather than building the infrastructure.
Hmm. Cloud computing expert here, too. I partially disagree.
I would simply say "it depends". For companies with specific use cases, e.g. Dropbox, it makes A LOT of sense to build at least 60-70% private, and the rest on AWS or GCP. Snapchat looks like a special case to me.
If you are doing just one or two things at a massive scale (Dropbox storing files), it makes sense. But, we are talking about Snap Inc, the infrastructure, and services it needs. A single high-speed intercontinental fiber cable costs 400 Million $ (https://techcrunch.com/2016/10/12/google-and-facebook-are-bu...). To build a service like CLoudML (Hosted TensorFlow), you need to write software (TensorFlow), build the ASIC Chips by manufacturing / by partnering with a chip manufacturing company like Intel / Samsung / Qualcomm, you need custom networking software and hardware to scale massively for data workloads (Andromeda), years of research to build these tools, you need to create your own programming language (Go and Dart) to be able to scale your development to these levels, you need to build data centers in multiple regions, you need to build DeeLearning models for cooling your data centers (https://deepmind.com/blog/deepmind-ai-reduces-google-data-ce...), ...
> if they focus on building infrastructure, it might slow them down
I always found this issue of "focus" a bit strange. Two billion can buy a lot of focus. For example, say there was a separate company smaller than Google that snap outsourced their infrastructure to, allowing them to focus, then they bought that company - it comes out to the same.
The amount of money on the table isn't 2 billion dollars. The amount at stake is the difference between 2 billion dollars over 5 years, and the theoretical best case operating costs of your own infrastructure, which very well might be higher than Google's because Google's scale is bigger. Snap might say to themselves that they don't want to be paying Google's margins, so they invest a billion dollars in development for infrastructure with identical performance costing only 350 million a year instead of 400 million. Is that worth it?
By "cloud computing expert", do you mean Amazon sales shill? It's weird enough that someone on HN would just use that vague credential of "expert" in the first place, but this thread is ridiculous ('round here, usually we recognize experts when they say something along the lines of "Hi, I'm the guy discussed in the article...", or "Hi, I actually wrote that software 15 years ago...").
You know that Snap can hire more than 3 people, right? They can have people working on the infrastructure at the same time that someone else works on "the product" they want to build. That's what happened at Google, and as a side effect, they now have a cloud platform they can rent out.
There are some benefits to using cloud services in some cases. It's rare that 100% cloud is a wise deployment move, especially for companies that operate at Snap-scale.
That's the point I'm trying to make. Snap has, say, 100 employees + $2b. The 100 employees are working on features. The $2b can now go to Google, or to purchase "focus" for in-house infrastructure, there is no slow down in developing features. Now if they can't technologically duplicate what they need using $2b is a different story which I have no opinion on.
They don't just have to build it though; they have to build it fast enough to handle their growth. $2B might not be enough (or it might not even be possible for any amount) to build what they need fast enough to not hamper growth.
Also, by buying from Google, they hedge against lower than expected growth too. If growth doesn't meet expectations, while they're still obligated to spend the $2B, they can re-sell the services for likely close to cost, given they'll be getting a discount.
Of course it's damn possible to build that with $2 billion. Will it be robust as google's infrastructure? No. Will it have (unlimited) scale as Google has? No. Will it have network connectivity and geographical distribution as Google has? No.
So, if you set scope on expectations and dedicate time and resources, it's definitely possible. Reasonable? Probably not.
> "We need to do X" and they say "Here is product Y that connects to product Z, but you need to write your own connector." And each product has it's own pricing structure (one by CPU usage, one with flat fees, one based on throughput) so working out total cost is a nightmare. Then you have to hire around those techs, project manage the build of the infrastructure, etc. At that point you start to ask "Why are we using them again?"
This is a feature(?) of AWS. Google Cloud provides fully built solutions for majority of use cases. AWS provides nuts and bolts and tell you to integrate them.
They don't stitch all that together for you. Nor do they have a good way of calculating how much each of those services will cost when used in conjunction with each other.
If you want a customizable solution, you connect the tools. Pub/Sub for streaming, Dataflow for processing data, Big Query for Data Analytics (Warehousing), Datalab is your development environment. Its hard for a single solution to serve 100% of customer needs. That why they suggest tools. Care to share what a better technology stack looks like? I would love to know if there are better alternatives for services like Google App Engine, Google Cloud Load Balancer, BigQuery, Pub/Sub, Dataflow, TensorFlow Cloud ML, Container Engine (Kubernetes), Stack Driver, ...
Speaking about cost, its all flat rate, with usage discounts (the more you use, the more discount you get) and there are no upfront payments / reservations / locking / hourly billings (like AWS)
> If you are looking for a solution for mobile analytics
I'm not at all, that example was pulled at random as an example.
> Its hard for a single solution to serve 100% of customer needs. That why they suggest tools.
It would be hard to describe just how custom of a solution my company would need. My company certainly likes to do things in a unique way. However there is a huge grey area between "here is a specific product that doesn't quite fit your needs" and "here are the blocks you need to build a custom solution on your own". That grey area is filled with money.
> I would love to know if there are better alternatives for services like ...
Don't underestimate the ability of large enterprise to roll their own. It isn't like Google is the only company that can build a message queue or a load balancer. Nor are they the only game in town selling them. This is a time to remember: The customer is always right. You and I may think that "Google is the best" but my manager wants what my manager wants. We may select a lesser solution simply because the support is significantly better. And that support should be vertically integrated. If we have support for pub sub, support for big query but not support for the integration between them ... that is a drawback.
> Speaking about cost, its all flat rate
Not really. For example data flow is based on the size of machine you are running (plus some extra percentage for the service), pub sub is based on number of events, big query is based on query time + egress, etc. I probably still have the spreadsheet I used to try to get a handle on costs kicking around somewhere. "Flat rate" does not describe the monstrosity that is that spreadsheet.
I totally understand that some people don't understand the enterprise mentality. I'm not trying to defend it - maybe just give a small view into it. Now that I live in it, I may not agree with it, but I understand it. Google doesn't.
> I'm not at all, that example was pulled at random as an example.
Then give me an example of what you are looking for and I will give you a solution
> My company certainly likes to do things in a unique way.
And every company has a unique way of doing things. That is why they provided all the tools you want and let you combine them to fit your custom needs. Instead, if they were to provide turnkey solutions, you would not use them because you have unique needs.
> It isn't like Google is the only company that can build a message queue or a load balancer. Nor are they the only game in town selling them.
If there are any better solutions, please let me know. Support is first class on Google Cloud. That is one of the reasons why Spotify choose Google Cloud over AWS.
> "Flat rate" does not describe the monstrosity that is that spreadsheet.
Google Cloud pricing structure is far easier to work with, as you don't need reservations/upfront payments and at the same time leaves room for scaling up and down elastically. The amount you pay is simply based on usage. For Pub/Sub its the number of messages, for BigQuery its the amount of data that is queried. I would love to see a simpler pricing structure. Would you care to share the details if you know of any?
A simpler pricing structure would be an Enterprise flat rate. Charge five to ten times what any possible scaling might cause. There are corporations that will pay. That's the point about "mind-bending".
Now that I live in it, I may not agree with it, but I understand it. Google doesn't.
I love the hubris of statements like this. Google makes decisions based on many factors, many of which you have no idea about. The idea that they're leaving an incredible amount of money on the table because they just lack all your enterprise experience is ludicrous.
> because they just lack all your enterprise experience
I can only speak to the architectural projects I have been involved with and to the general attitude around the office. I could rephrase my last statement as: "The general impression from the technical managers at my workplace is that Google does not demonstrate understanding of the factors that influence the types of large enterprise purchases we routinely make".
I might suggest you are mistaking my description of the inner-workings of my workplace with a value judgement, or with my own technical assessment of Google's technology. What you need to understand is that it is people like me pitching Google Cloud and witnessing it getting shot down. I'm trying to explain to you why.
What you need to understand is that it is people like me pitching Google Cloud and witnessing it getting shot down
And you feel you have sufficient understanding of all the tradeoffs involved in Google choosing to implement what you want them to implement in the way you want them to implement it? You know enough about their overall Cloud strategy, their future development plans, their staffing situation, etc?
Because it sounds like you're saying, "I have some money I'm willing to pay. Google won't take it, Google just doesn't understand!"
There are lots of jobs you could offer me that I would turn down for a variety of reasons.
I have a perfect example for how little Google understands Enterprise.
This issue[1] affects almost every enterprise developer using this product (the original AppEngine) and has been open since 2008 (Nearly 10 years!).
I provided a (awful!) work around in 2010[2] for the Java version, and there are similarly dreadful hacks for the Python version. It would be trivial for Google to implement one of these fixes.
Ironically, that's the way that Microsoft used to operate. They were very big on providing the platform and API, then insisting that partners built the solutions. That was back in the Gates and Ballmer era.
I guess that they've learned from that major mistake.
I worked with both Google Cloud and AWS. Having read Gartners Magic Quadrants on Cloud, I really question Gartners competency on understanding what Cloud really means. There is no question that Google Cloud is years ahead of AWS (ease of use, scalability, performance, vertical integration, networks, security ... ), but Gartner reports that AWS is leader. Bottomline: Don't trust what others say. Try out various cloud providers and decide for yourself, before diving head down and suffering with pain everyday.
That MQ is led by Lydia Leong who is probably the most respected analyst in cloud (dating back to 2008!)... they're pretty transparent in the actual paid report.
AWS Lambda has a major head start and is a big reason for its head of vision above Google. Don't get me wrong, Google has a great quality cloud, but the pace and breadth of Amazon's offering is breathtaking.
One of the big controversies was IBM Softlayer being reduced to niche status, which led to howling of a volume never heard out of New York... which delayed the report months... but Gartner stuck to their guns. It's a pretty fair report.
I am AWS heavy and used Google Cloud a bit. There are tons of AWS services where Google just does not provide an equivalent. Also, AWS release pace is incredible.
Quality of Google Cloud is very decent though.
Google Cloud and AWS are pretty much equivalent in terms of feature parity: https://cloud.google.com/free-trial/docs/map-aws-google-clou.... There are few (10% os services maybe?) that are not available on other platform. Also, the way AWS counts service is by nuts and bolts (AWS believes in providing fundamental components, not higher level frameworks) vs Google Cloud counts things by Cars / Trains / Submarines levels. That why people tend to think that Google Cloud provides fewer services. (As an example, imagine how many number of AWS components are needed to perform the function of App Engine. May be 10?)
>> Having read Gartners Magic Quadrants on Cloud, I really question Gartners competency on understanding
I think you can replace "Cloud" with whatever technology Gartner or most analysts write about.
These firms are little more than uninformed hype producers that fuel the enterprise software spend cycle through marketing (webinars, "research", tap-my-back-events and 3k€ "reports") that tell CIOs/CTOs what they should already know, by definition of their function: what technologies to consider for their business problems, why and how.
On the other hand, they provide a nice scapegoat when someone needs to offload responsibility, a bit like the "no one ever got fired for buying recommended by Gartner".
PS: wish I could find Zed Shaw's video on "calling bullshit" (or related), as it includes a nice anecdote on dealing with industry analysts.
As someone who works with technology every day, i never have any idea what Gartner is trying to say. I look at their magic quadrants or whatever, and have no idea how it is supposed to make decisions about technologies or their vendors.
I am very happy to let you know that perspective is informed and correct too! Google Cloud is far better than AWS. Especially in things like Ease of use, scalability, big data, machine learning, NoOps, pricing Google Cloud beats AWS by miles margin.
Agree 100%. For instances, you have standard, high memory and high CPU and custom. Super easy to make sense of rather than crawling through webpages to understand what each type of instance mean. For example: AWS has r, t, m, g, p instance types. When you look at the whole platform, naming, gotchas etc AWS creates lot of cognitive overhead for developers. I find Google cloud far easier to use, partially due to things like these.
> Many of these solutions are unavailable below a certain scale
This was true before Google Cloud. With Google Cloud, you can enjoy these benefits whether you are an individual developer with a sub 100$ monthly budget / a mom n pop shop with 1000$ budget or a SMB / Startup with a 10K$ - 100K $ to spend on your infrastructure.
That's exactly what he's saying. Small startups, who don't want to use Google, would have a lot of difficulty implementing some of these designs on their own.
Here is how Google Cloud achieves serverless. Ingesting Data: PubSub. Scales to millions of messages instantaneously, no need to spin up and spin down the capacity/shards (like as in Kinesis), exposes RESTful interface for ingesting messages from anywhere (web/mobile ... ). If you want a high-performance interface, you get gRPC as well. With RESTful interface to consuming messages, you can connect PubSub to anything you want or trigger Cloud Functions / write to storage / ingest to Stackdriver (Monitoring system) using managed services.
Querying Data: Big Query. No need to spin up a single server. Just write your query in SQL and watch the magic of a thousand servers being spun up to serve the query in a fraction of second and process a petabyte of data in a couple of minutes. btw ... you can ingest data into Big Query in real-time and analyze results within in seconds.
Process Data using Dataflow: For workflows that are more complex than SQL queries to ones that need to be running continuously on streaming data. Dataflow is the serverless version of Spark. No more running our of memory errors, much fewer hassles with hotkeys, no more manual performance tuning of buffers, no more cleaning up log folders. If you are using Spark, but have not tried Dataflow, you are missing some serious magic.
Google Cloud ML for Machine Learning: Cloud ML is a hosted solution for running TensorFlow jobs. No more manual hyperparameter optimization, no more spinning up GPUs, no more scaling up the cluster size. It's all taken care for you.
Container Engine: Hosted version of Kubernetes. I am sure, everyone is aware of what K8 is and its capabilities.
I am from a Data / Analytics / ML background. The above was my reality of Serverless since 2015. Google Cloud has serverless options for Web (App Engine) / Mobile (Firebase), and other purposes as well.
Having done PhD in Cloud Computing and Big Data and I don't find Infrastructure/Big Data a sexy problem anymore. To a large extent, it's a solved problem. Building a data platform with a team of 3 people that can handle 10+ petabytes of data is easy. What is on the horizon and unsolved yet, is AI!
Would love to see if there are any other better/compelling Serverless options.