How does bookmarking work/How do I keep track of how far I've read while replaying from Batch? Will you also index by date? It can take a long time to replay a lot of data; do you have any numbers on the read rates you support per topic?
> How does bookmarking work/How do I keep track of how far I've read while replaying from Batch?
We do not have any bookmarking functionality built (yet) as we currently expect folks to just tweak their search query. Each one of the events has a new id attached to it that you can query and reference during search.
> Will you also index by date?
We do! Every event has a microsecond timestamp attached to it.
> It can take a long time to replay a lot of data; do you have any numbers on the read rates you support per topic?
We've done some initial replay throughput tests and have been able to reach ~10k/s outbound via HTTP - of course, this is all _highly_ dependent on where you're located. We expect that for folks who need super high throughput, we'll probably need to be closer to them - we fully expect to have to peer with some of our customers and optimize for throughput by doing gRPC and ... batching :)
So far, we've done most of our testing on inbound and we are currently able to sustain ~50k/s (with ~5KB event size). Our inbound is able to scale horizontally and so can go waaaaay beyond 50k/s if needed.
We have a ton of service instrumentation so we've got good visibility around throughput (and thus should know well in advance as to when we're starting to hit limits).
This interview with Kirshner is a good read as well: https://www.aip.org/history-programs/niels-bohr-library/oral... And a reminder that science is not generally moved forward by strokes of brilliance, but by a whole bunch of people grinding at the fuzzy edge until the picture becomes a little more clear.
Great quote. Also unintentionally (I think) humorous, given the manufacturing problem with Hubble referred to in the article was slightly inaccurate grinding of the lens' edges.
23AndMe, which is located in California, claims on their website that they are required to maintain any information from genetic tests indefinitely. How is Probably Genetic able to 'destroy samples and data' on more than a soft-delete basis?
Apologies for not getting to this earlier. Our in-house GC is est fit to answer this; however, as far as I am aware, CLIA only requires that we keep samples until the analysis is complete and results are returned to patients. They recommend that one keeps this as long as possible. The other thing is the difference between a report and raw data. I believe that we are supposed to hold onto reports for 21 years, some states have slightly different regulations but that's different from raw data. We are also a physician-ordered LDT while 23andme is FDA approved so there are differences there as well.
So it's important to note we don't test for diseases but rather variants which may be the etiology. I'm not an expert in this condition by any means but a quick search says we still don't know the candidate genes for this and many have been excluded which would be problematic for our test.
Walmart - Chat/Automation | Hoboken, NJ and Bangaluru, India| Full-time | ONSITE | VISA | contact me directly cole.dutcher at walmart.com
Looking for: engineering manager and all levels of engineers, F# team but no experience necessary
I'm looking for an engineering manager and engineers of all levels to help us build automated customer care solutions over chat, sms, smart speaker, and phone. We have built a strong automation platform, and this year is dedicated to creating industry-unique capabilities and leveraging them across the highest-impact brands and verticals in the Walmart portfolio.
Some things we believe in are:
1. High expectations and autonomy
2. Libraries over frameworks
3. Event-Sourcing/CQRS
4. Functional programming
5. Uptime & performance as a feature
6. Continuous learning and sharing
7. Data-driven decisions
8. Respect for team members
9. No. 8 Allows for vigorous discussion of ideas
If this sounds like a fit, please reach out to me directly.
When I read "design docs that stay up-to-date" I thought you meant system architecture docs that integrate with AWS, Azure, GCP, DBs, etc. I hope someone builds that.
