Yeah, this is really, really far from an apples-to-apples comparison. First of, the test dataset size is trivially small for usecases where big data systems are typically applied. I don't know why you'd introduce all the complexity and overhead of a distributed mapreduce framework to ETL a dataset that would fit in memory on consumer-grade hardware. It's not exactly fair to compare a framework running on a single node to one where you've artificially introduced multiple nodes and network overhead for a dataset that would easily fit on one. You'll also notice a pretty stark difference between the level of detail provided for the BlazingSQL test set up and the Spark one, which (unless I'm missing something) is lacking any code or configuration details. I've dipped my toes in the big data space long enough and seen enough "${FANCY NEW FRAMEWORK} beats ${INDUSTRY-STANDARD FRAMEWORK} by 123x!!" posts to recognize this as a gigantic red flag. How you manage partition sizes, order and choice of operations, and tuning parameters can make orders-of-magnitude level differences to your performance.
Maybe the future of frameworks like this will be on the GPU. I'm just not seeing any evidence of it yet. Right now, Spark fills the space where you can throw globs of memory at TB- to PB-scale problems. I could very well be wrong, but I don't see how this is going to be cost-effective on GPUs given the current cost of memory there.
1. You don't have to fit your whole workload on the GPU you can process it in batches like you would for a workload that doesn't fit into memory on a non gpu solution. You don't need PB of GPU memory to run PB workloads.
2. The dataset is trivially small because this is a new engine built for the rapids eco system and it is limited for the time being to a single node. We are releasing our distributed version for GTC (mid March) and will be able to give you more reasonable comparisons. This is a similar path of development to our pre Rapids engine which went from single node to distributed in about a month because we have built this engine to be distributed. Right now we are finishing up UCX integration which is the layer we will be using to communicate between all the nodes.
3. You can always try it out. Its own dockerhub (see links in this post) and if you want to run distributed workloads right now you can manage that process using dask by handling the splitting up of the job yourself. In a few weeks you will be able to have the job split up for you automatically without need for the user to be aware of the size of the cluster or how to distribute data across it.
We're pretty excited near-term for getting to sub-second / sub-100ms interactive time on real GB workloads. That's pretty normal in GPU land. More so, where this is pretty clearly going, is using multiGPU boxes like DGX2s that already have 2 TB/s memory bandwidth. Unlike multinode cpu systems, I'd expect better scaling b/c no need to leave the node.
With GPUs, the software progression is single gpu -> multi-gpu -> multinode multigpu. By far, the hardest step is single gpu. They're showing that.
1. If you process it in batches then you have to count the time it takes to send the data of each batch to and from the GPU.
2. It's fair to start out with small data sets, but then you don't compare against distributed frameworks like Spark, but rather against single-node solutions.
Also - Spark is very slow compared to analytic distributed DBMSes.
I used to think that experience meant the difference between believing benchmarks and being skeptical of them. Now I know it's the difference between being skeptical and ignoring them.
What business does this guy have deciding what does or doesn't empower women in tech? Where is his evidence or personal experience here? I'm certainly no expert in this area, but couldn't it be better listen to the opinions of women in tech, disturbingly many of whom who have publicly shared stories of sexism or harassment directly limiting their career potential in tech, about whether they feel underrepresented because of interpersonal culture or because of their biology?
You don't need personal experience to have an opinion, if you are capable of reading what other people have written and bringing to bear your own experiences.
It does not mean you get to decide anything for others, but it is a basic sign of respect to listen to what people say, and it is an even more basic sign of respect to not attempt to defame them.
This guy clearly isn't perfect. He was naive about what the effect of releasing his memo would be. However he was respectful and seemed to be trying to temper his perspective. He doesn't deserve the raw hatred and condemnation he is receiving. I think we should be compassionate and try our best to tolerate our differences.
I applaud the guy for doing research and putting together an argument. He knew the backlash that would come from it. I think he thought that the people at Google would respond fairly reasonably. Now that he has been fired, people will use that to point out the intolerance of the people in the Echo chamber..thus proving his point.
They should have included him on the diversity and inclusion leadership team. Sad that Google went the route they did.
This is an extremely different takeaway from mine. From what I read, the author repeatedly made it clear that he is not stating that women or racially underrepresented workers at Google are individually worse at their job. In fact, most of his discussion on different tendencies in gender are focused on selection of job field type of job within the field rather than ability. Furthermore, the author repeatedly reiterates that these tendencies are just that: tendencies, not rules. Some women pull 70+ hour work weeks. Some men work part time. To say that a larger portion of one group, even in the absence of cultural and societal pressure, is going to choose either the former or the latter is not disparagement.
To put this in a different context, take the gender discrepancy in murderers. Across nearly all countries, men commit roughly 75-85% of murders. This nearly uniform across agrarian, industrial, and post-industrial countries, rich and poor countries, liberal and conservative countries. I would consider it totally fine to say that men innately have a greater tendency to commit murder than women (and for what it's worth I'm a man myself). Is it okay to say that because I have a Y chromosome I should be treated as a murderer? Or that I should face a different standard of evidence if I'm on trial for murder? Of course not. But stating that I, as a man, am statistically more likely to commit murder than the average woman is not the same as saying I should be presumed to be a murderer and it is not the same as saying that I should face a different standard of evidence. But the point remains, it is okay to conclude that the discrepancy in murder rates is because of innate tendencies rather than discrimination by police, juries, etc.
I wish the guy had written what other people had written. This stuff has been hashed, rehashed, refried, deep fried... and he got all the biology research wrong. It just seems that he did no research, instead thinking about his feelings a lot and claiming he got somewhere original without engaging the existing scholarship at all.
Agree that it's a basic sign of respect to listen to what people say. I am tired of guys not listening to me saying I'm a boring, normal woman who likes math/tech/computers, not a freak or a biological anomaly, some evolutionary mistake.
> And he is not saying that you are a "freak" or "biological anomaly", he is talking about aggregated, statistical differences.
He provides no evidence to back the conclusions he draws from that data though. It reminds me of some articles or posts you might see briefly gain traction on the internet where the author starts from a point backed by one or two reputable sources and proceeds to use those references to back an argument that lies well outside the bounds of the original data.
As one of the authors of a paper cited in the memo puts it:
> In the case of personality traits, evidence that men and women may have different average levels of certain traits is rather strong. [...] But it is not clear to me how such sex differences are relevant to the Google workplace. And even if sex differences in negative emotionality were relevant to occupational performance (e.g., not being able to handle stressful assignments), the size of these negative emotion sex differences is not very large [1]
The post I responded specifically said the memo author got all biological research wrong. Now, his conclusions from that are certainly more debatable.
I guess the larger point I was trying to make is that when people feel singled out because of statistical differences b it leads them to take things personally, which leads to perpetuating disproportionate reactions and a non-existing platform for actual discussion.
Vets are 80% female. Is the industry closed to men that might want to be a vet? Do we need more outreach programs for men to be vets?
Or maybe is it that men are less likely to want to medically treat animals than women, and the gender discrepancy is one of preference? And yes, men who do become vets are statistical outliers, but certainly not "freaks". It's likely the same for tech and women - you are a statistical outlier, and from your perspective things are the norm, but the majority of women are not interested in the same things that you are interested in professionally. (The majority of men aren't either, but there's a larger minority of men sharing your professional interests than a minority of women).
And as for "not getting somewhere original" - the author's point about the difference between comparing averages vs the overlap of bell curves around the averages is a great point and not one I'd seen highlighted that way before. Perhaps you have, but it was novel to me.
80 years ago all vets were men. How can men's biology change so fast? 150 years ago elementary school teachers were men. In 1984 more than 35% of CS majors were female. Not long enough for genetic change.
> Agree that it's a basic sign of respect to listen to what people say.
> This stuff has been hashed, rehashed, refried, deep fried... and he got all the biology research wrong. It just seems that he did no research, instead thinking about his feelings a lot and claiming he got somewhere original without engaging the existing scholarship at all.
Please provide examples. Your statement is incorrect. Looks like you didn't bother to listen correctly to what he said.
He may have not told his direct teammates they lowered the bar but he certainly implied that women and underrepresented groups have at a whole at Google:
> The Harm of Google’s biases:
...
> - A high priority queue and special treatment for “diversity” candidates
...
> - Hiring practices which can effectively lower the bar for “diversity” candidates by decreasing the false negative rate.
Do you really not see the concerns this engineer has introduced for Google over his ability to objectively judge his peers who are women or members of an underrepresented group or at least how such coworkers might now have these concerns if they find themselves working with him? Do you not see how the long-term detriment and risk to Google could be (substantially) greater than the value the engineer brings to the company? Even if you remove the high-mindedness and virtues that tech companies like to project in their PR, do you not see how this is the only rational business choice for a Fortune 50 company?
There is something I don't understand. The third quoted line should raise an entirely different question.
You are in the position where you want to constantly hire a lot of qualified candidates. If you have discovered a way to decrease false negative rate, why would you not apply that across the board? It's like you have learned how to fix hiring but then carry it out only with some arbitrary cohorts.
My guess is maybe because it cost too much to implement it across the board, and the side effect that focused application results in more minority/women hires.
> fired from a tech company for "perpetuating gender stereotypes"...
Fired from a tech company for publishing a company-wide memo insinuating that some of his coworkers are, on average, at a biological disadvantage for the type of work they do and suggesting that some subset of them haven't achieved their position based on merit.
And listed data and studies to back up his claim. That it is controversial, does not mean it isn't true, and you haven't refuted it, simply stated it as if on it's face should have been grounds for dismissal.
All we really know is that the Silicon Valley thought bubble, and political intolerance is extreme to such a degree that people have to watch what they say and think at all times so as not to anger the thought police. The moral superiors.
No room for discussion. No room for debate. Just fall in line and be sure to advertise your virtue and 100% agree with views that the political left mandates you to hold.
> And listed data and studies to back up his claim. That it is controversial, does not mean it isn't true, and you haven't refuted it, simply stated it as if on it's face should have been grounds for dismissal.
He listed references that indicate, when viewed broadly, that men and women can exhibit different observable personality characteristics. That was not the argument of his memo. His argument was that, because of this broadly measurable difference, this leads to women being less interested in computer science (although not to the degree of similar STEM fields, oddly enough!) and therefore he shouldn't have to be subjected to bias training or be burdened with any other sort of program that suggests maybe, perhaps, potentially the social culture in tech/SV could be exacerbating this.
> All we really know is that the Silicon Valley thought bubble, and political intolerance is extreme to such a degree that people have to watch what they say and think at all times so as not to anger the thought police. The moral superiors.
Did the thought police make this guy write and publish a work memo suggesting some women are too neurotic to be software engineers?
> No room for discussion. No room for debate. Just fall in line and be sure to advertise your virtue and 100% agree with views that the political left mandates you to hold.
How about don't write a company-wide memo that insinuates some of your coworkers aren't there by merit because they don't have the biological composition to stomach the job? It's not like somebody was having a conversation with this guy in the break room, asked his opinion on diversity and ran to the press to start a witch hunt against some random Google engineer.
> That was not the argument of his memo. His argument was that, because of this broadly measurable difference, this leads to women being less interested in computer science (although not to the degree of similar STEM fields, oddly enough!)
I mean, he titled the section of the document that talks about those effects "Possible non-bias causes of the gender gap in tech." But hey, those are sinful ideas, so he must have been a sinful man to have even entertained them in the first place.
Listed "data" and "studies" doesn't suddenly make you right. Studies are far from perfect, especially in this field, and even more so when tons of other studies contradict these conclusions. You don't get to wave a magic want and say "I'm using FACTS so you're wrong!!"
> insinuating that some of his coworkers are, on average, at a biological disadvantage for the type of work they do
That's not what he actually said. In fact the memo had a chart and a paragraph or two specifically stating that taking average traits of a group of people and applying them to individuals was wrong and misleading due to large overlaps of traits between groups.
He was not implying anything about his colleagues that are already working for Google, rather he was pointing out potential biological differences in averages at the population level that possibly cause fewer women to enter tech (and hence making it difficult to get an even ratio of male/female employees).
There has been absolutely zero concrete evidence presented that indicates it was Seth Rich. I fail to see how this is anything more than a conspiracy theory being promoted by members of the right wing media in order to distract from the current political attacks against Trump at the expense of the family of a man who was senselessly murdered.
Julian Assange is offering a 100K reward for information about the murder. A PI hired by the Rich family has stated publicly the police were told to stand down and that sources available to Fox News told him Seth Rich had been communicating with wikileaks. Kim Dotcom -- well you can click the link at the top of the page yourself.
I understand this is all circumstantial evidence and hearsay, but I would like for you to explain which of these people are "members of the right wing media."
Basically you're telling me that all of these people are conspiring to distract from Trump conspiring with Russia. I'm pretty sure you have the conspiracy theories today.
>A PI hired by the Rich family has stated publicly the police were told to stand down and that sources available to Fox News told him Seth Rich had been communicating with wikileaks. Kim Dotcom
This has since been retracted by Fox since it wasn't true and was made up by the reporter and the PI.
Wheres the comment by Fox saying that the PI made this up? Their page last time I checked just had a sentence saying it did not meet their level of scrutiny or something along those lines.
I see it more as Assange distancing himself so that nobody later says he outed a WL source in case the allegations turn out true. If anything, the Nieuwsuur interview in August was more of a wink wink but still hardly qualifies as evidence.
Yes, hardly, because there is little chance that Assange is going to confirm this.
We don't even know for sure if he actually knows something or simply pushes for investigation of this murder solely because other leakers are worried that this conspiracy may possibly be real and that the same could happen to them.
Or because his room in the Ecuadorian embassy is too small and he would prefer to live in Moscow with Snowden /s
> Steve Jobs was brilliant at coming up with new products
By my count, Jobs shepherded the following hit products (I'm giving the Apple ][ to Woz):
- Macintosh (1984)
- iPod (2001)
- iPhone (2007)
- iPad (2010)
I think you're exaggerating how often Apple rolled out new product lines under Jobs. You could even argue that the iPod and iPhone overshadow the other two substantially in terms of cultural impact.
> Their only option now is to start creating variations of existing product lines to keep them "fresh".
Isn't this kind of what you'd expect out of a business? Refreshing and supporting their existing product lines while occasionally trying to establish new ones? The iPhone is looking more and more like a once-in-a-lifetime home run product (Ben Thompson has done a lot of good writing on this), the likes of which Apple (or anybody else, for that matter) may never be able to top. This isn't to say that Apple doesn't miss having a visionary like Jobs at the helm or that their product lineup doesn't have glaring flaws. I just think the massive success of the iPhone set expectations way too high.
Maybe I'm misinformed, but those seem like the "definitive" Apple products to me. They're basically the foundation of their current business, and have only been diluted since his departure.
Why would Jobs not get credit for the Apple 2? It's not like he engineered any of the other items on your list. It was his marketing that made the Apple 2.
When Jobs came back to Apple he supposedly canned hundreds of different products and got them to focus again. While he was around, that's how things remained; each product had their own place. As time goes on since his departure, the clutter is starting to come back and the lines are starting to blur. Whats the difference between the new Macbook and the Macbook Air? The iPad 9.6 vs the iPad Pro 9.6? Apple Watch Gen 1 vs Gen 2 vs Series 1 vs Series 2?
The most concerning thing to me is that after recognizing the parallels between the Trump campaign and the Silicon Valley ethos Altman's response wasn't to be introspective and consider if SV culture has been too boorish but rather to (seemingly) pay compliment to Trump's craftiness.
Android solution: You own the server. Your server pushes directly to you.
iOS solution: No ability to maintain SSH connections, instead you need to get a 3rd party app to open an SSH connection for you and send you a push notification via an Apple service when a change occurs.
See the problem? Trusting Apple alone isn't an option - Apple doesn't offer a service to send you a push notification for your SSH session. You need to involve a 3rd party.
That's the way you see it, how about the way I see it?
iOS solution: everything goes through Apple, I generally trust them and know that they try to look out for me.
Android solution: every app goes through someone else's server, every app needs individual vetting because I have no idea what they're doing or how hard they're trying with their privacy. Where is the trying at all. That's the way you see it, how about the way I see it?
iOS solution: everything goes through Apple, I generally trust them and know that they try to look out for me.
Android solution: every app goes through someone else's server, every app needs individual vetting because I have no idea what they're doing or how hard they're trying with their privacy. Or if they're trying at all.
I like the iOS solution. You have it framed for my developer point of you, and I can understand that. But from your point of you, your solution has a lot of potential issues around privacy alone.
Apple looks out for their bottom line. If your interest happens to cross that path, bummer.
Don't just assume a commercial interest 'looks out for you' because they don't. If your needs, or at least your perceived needs (due to effective marketing) fit with a company's strategy you might get the idea that said company does it all for you and your fellow users. This does not imply any causality, the company just does what it thinks is best for its bottom line. As long as a company serves the needs of its users they are in a position to do well. Vendor lock-in serves companies in that they have more freedom to choose their own path without bothering all that much about whether that path coincides with their users' needs as those users face a steep cost if they choose to change vendor. As long as the company keeps the extra costs - in money, limited features or usability - lower than what it would cost a user to switch vendor they stand to gain from this strategy.
I mean, what he said is true. Look how many rogue apps on Android start spamming you with notifications and even worse, apps that push notifications that are purely advertising. Since I switched to iOS this hasn't happened once, and I'm prompted on first use of the app if I want to allow notifications or not.
You honestly believe that Google don't look out for their bottom line either?
You honestly believe that Google don't look out for their bottom line either?
I did not mention Google, nor any other company (other than Apple) for that matter. Instead I used the words 'a commercial interest' to indicate this goes for _any_ company which exists to make money - and that means nearly every company in existence.
But, to get back to your Google example, of course Google looks out for their bottom line. It just so happens that Google is not nearly as aggressive in herding their users into fenced corrals as Apple is. Google moves fast and often discontinues services so it is unwise to base your (company's) future on the existence of any specific Google service. Fortunately Google generally makes it easy to get data out of their services so a viable exit strategy is usually feasible. Apple is not that fast a mover which also implies they don't discontinue services at the rate at which Google does. Where Apple fails miserably is in their support for migrating data out of their services. This, again, fits the description I gave at the start of this sub-thread: Apple wants to raise the cost of leaving their services.
For the most part thought the Android solution matches the iOS solutions there though, devs use GCM. You're however missing the part where app devs have their own servers that have to talk to Apple or Google. You have to trust those 3rd parties too, all Apple or Google does is get notifications from their servers to your phone.
I'm not talking about general push notifications though, nor am I speaking as a developer. I'm speaking as a user, who also wants to connect to ssh, imap, xmpp and sip. The standard Apple/Google model is fine for most push notifications but absolutely not for things like this where a direct connection to a server is something that you as a user want.
That's the part I'm complaining about, not the general push message system, but the system of not being able to keep a persistent connection to a SIP or IMAP server without giving my credentials to a 3rd party so they can handle push and forward that off to Apple or Google.
> Yes, there's an argument that there is very little user benefit from having multiple push notification services. But there's no inherent benefit to having Apple's service be the one-and-only service ever allowed to be used -- that's not an opinion, that's vendor lock-in.
Let's suppose Apple were to open up their APIs to allow other services to send push notifications to their devices. Since they are no longer the intermediary they've given up control on how often a developer's servers are sending notifications via the third party, they've lost a control on throttling how often notifications are sent to devices. What happens if the push service or the developer's backend is compromised (or misconfigured) and attempts to send hundreds of push notifications per second to thousands of phones? Features could be added to the OS to help mitigate this but the phone must still service the incoming traffic. Strictly speaking, this isn't Apple's fault but this won't be a sufficient answer for their customers.
Which isn't even considering that they need to figure out a secure mechanism for allowing third parties to uniquely identify iPhones.
> There's an argument that there is benefit in not having multiple different web browsers on a user's phone. But there's no benefit to having Apple's webkit be the one-and-only web browser ever allowed on the phone -- that's not opinion, that's vendor lock-in.
What happens if Mozilla, Google, etc. release a new browser engine for iOS with all the latest whizz-bang HTML features but at the cost of doubling power consumption? Apple support needs to deal with the customers who suddenly see their battery life plummet when switching to a new browser. 99 customers out of 100 will blame Apple for this. What if a vulnerability allowing RCE is discovered in the JIT code included in the third party browser? Apple has lost control of the patching cycle and their only recourse would be to remotely remove the browsers from their customers' phones if the vendor does not update the app in an acceptable timeframe.
I feel like a lot of Apple's decisions around the iOS ecosystem can be explained by looking at the problems they had with customers running Adobe Flash on the Mac.
> What happens if Mozilla, Google, etc. release a new browser engine for iOS with all the latest whizz-bang HTML features but at the cost of doubling power consumption?
What if I use an email app or notes app or other app that consumes twice the power of Apple's inbuilt app?
I don't see what's special about a web browser. It's just another app.
Maybe the future of frameworks like this will be on the GPU. I'm just not seeing any evidence of it yet. Right now, Spark fills the space where you can throw globs of memory at TB- to PB-scale problems. I could very well be wrong, but I don't see how this is going to be cost-effective on GPUs given the current cost of memory there.