Working on new backend for Mercury framework. Mercury makes it easy to serve Python notebooks as web apps. You dont need to know HTML, CSS, JS to build beautiful looking web app from Python notebook. I'm writing new backend to support ipywidgets, anywidgets in Mercury. https://github.com/mljar/mercury
Few questions:
1. Can it work with tabular data, images, text and audio?
2. Data preprocessing code is deployed with the model?
3. Have you tested use cases when ML model was not needed? For example, you can simply go with average. I'm curious if agent can propose not to use ML in such case.
4. Do you have agent for model interpretation?
5. Are you using generic LLM or have your own LLM tuned on ML tasks?
1. Tabular data only, for now. Text/images also work if they're in a table, but unfortunately not unstructured text or folders of loose image files. Full support for images, video, audio etc coming sometime in the near future.
2. Input pre-processing is deployed in the model endpoint to ensure feature engineering is applied consistently across training and inference. Once a model is built, you can see the inference code in the UI and you'll notice the pre-processing code mirrors the feature engineering code. If you meant something like deploying scheduled batch jobs for feature processing, we don't support that yet, but it's in our plans!
3. The agent isn't explicitly instructed to "push back" on using ML, but it is instructed to develop a predictor that is as simple and lightweight as possible, including simple baseline heuristics (average, most popular class, etc). Whatever performs best on the test set is selected as the final predictor, and this could just be the baseline heuristic, if none of the models outperform it. I like the idea of explicitly pushing back on developing a model if the use case clearly doesn't call for it!
4. Yes, we have a model evaluator agent that runs an extensive battery of tests on the final model to understand things like robustness to missing data, feature importance, biases, etc. You can find all the info in the "Evaluations" tab of a built model. I'm guessing this is close to what you meant by "model interpretation"?
5. A mix of generic and fine-tuned, and we're actively experimenting with the best models to power each of the agents in the workflow. Unsurprisingly, our experience has been that Anthropic's models (Sonnet 4.5 and Haiku 4.5) are best at the "coding-heavy" tasks like writing a model's training code, while OpenAI's models seem to work better at more "analytical" tasks like reviewing results for logical correctness and writing concise data analysis scripts. Fine-tuning for our specific tasks is, however, an important part of our implementation strategy.
Thanks a lot! On a side note: big fan of mljar here. When we were initially playing around with using agents for automating ML tasks, we had used problems from the openml's automl benchmark which you had posted about on Reddit for our initial tests
If you are looking for Python notebook that is beginner friendly, I'm working on desktop app called MLJAR Studio. It packs Python and JupyterLab in single executable - installation is stright-forward. Additionally, there are extensions that makes working with Python much easier for beginners: for example, variable inspector and packages manager.
I think you can easily load JSON data into Pandas DataFrame. Then you can create visualizations and compute statistics. Python might be usefule for that.
... and if you (the parent comment author) wants a really easy tool for creating a UI for the Python code, I'd recommend looking at Streamlit: https://streamlit.io/
Also, much less expensive to implement. Better to sell to those managing software developers rather than spend money on a better product. This is a tried-and-true process in many fields.
We write and run tests to build trust in our code changes. But maybe tests aren’t the only way to achieve that trust.
When I was younger, I had a friend who was a senior software engineer. I remember he would make changes to production systems without even running the application locally or executing any tests, and yet his changes never failed. The team had a high level of trust in all his code changes.
The book that really got me hooked was The rise and fall of the Third Reich, by William Shirer. He was a journalist in Berlin before WWII, and you can see the journalistic background shine through in his narrative. A lot of what I've read since then has been in trying to trace back the references he gives to how the events of WWII recapitulated older conflicts, which I then went and found references in which to learn about.
Barbara Tuchman is a delightful mostly medieval historian, and Mary Beard is a wonderful Roman historian. Both have a decidedly personal voice, which happened to click well with me; your milage may vary.
Someone else in the thread mentioned Simon Sebag Montefiore, and I've enjoyed most of what he's written. It feels a little less scholarly than some other popular historical sources, but that's just an informal impression from a tyro.
Lots of people enjoy Yuval Harari, but I do not enjoy him at all.
IMO it's too broad. You need to have something you want to learn more about and then try to find a couple of different books on the subject to get a clearer picture.
My interest grew when I had the realization that you can easily spot "gaps" in history, if you just start asking "Why?"
Example: This country invaded that country. Why?
History is full of cause and effect. The fastest way to figure out where you're not getting the whole story is if there's not a "Why?" or the reason simply doesn't make sense.
I enjoy US history, but I've learned to gravitate towards topics that simply don't get discussed much in the "highlights", with the highlights being the American Revolution and WWII. Just for example, the War of 1812, French and Indian War aren't talked about much but are pretty interesting. Learning about the Cold War by reading Skunkworks was really cool. There's a book called The Code Book that is a great read and covers a lot of "history of encryption" during wartime that I really enjoyed.
On the Biblical side of things, I went down a rabbit hole trying to track the migration of the Jewish Tribe of Dan from Cyprus and their influence on Greek/Roman mythology. Just really interesting stuff to me so I even blogged about it.
A people's history of the united states is a good one. Most history is written by the generals and leaders. This account is colored by the oft ignored people.
and this history texts are provided as a counter balance to the biased sources and failure to examine opposing views of other history texts.
take in a range and make informed decisions. When a book presents any factoid, something else was chosen to not be presented. And that choice colors the narrative. Those who say "teach the facts" miss that the facts chosen paint a bias.
You basically say that everyone is biased, take the middle.
But I've seen with my own eyes living in dictatorial countries, that there are wild differences in how much bias there are. If one source lies 5% of the facts and an opposing source lies 95% of the facts, taking the middle leaves you with 50% lies, heavily biased to one side. You'd better just go with the first completely and have only 5% lies.
I've read several real scientific historical papers, and I immediately see they don't work like that. They are typically very-very boring, with tons of views and sources mentioned, considered and scrutinized. In fact, that's a good test for how scientific is a paper -- real scientist knows about all the works in their field, so omission of anything is a red flag. So they really strive to include everything available.
One history says Christopher Columbus sailed across the ocean with his three ships and made friends with the Indians.
Another history says, with quotes from him, that the people he found are very friendly and will make great slaves.
I'm not saying take the middle. I'm saying there is a narrative. Who's story are you hearing and why? The usual narrative we get supports those with power. Who's narrative is suppressed and why?
You get a narrative about Columbus the explorer and bringer of civilization to the new world. The other narrative is that he brought untold suffering and despair. Did he expanded European civilization and manifest destiny, or did he murder and enslave and subjugate people and their children to the horrors of chattel slavery?
Which one is a lie? Which is fact or not fact? It's both. It's neither. It is the narrative that is painting the story that is being told.
> so omission of anything is a red flag
you would then agree that all the factoids and accounts presented in A People's History of the United States _not_ being present in other works and education of our students is a red flag. But, honestly, and you know this, you can't include _all_ the facts and stories. By picking something, something else is not included.
You pick a photo to share of your family smiling, you didn't pick the photo of them fighting. Is your photo a lie? Heck, you may not have even thought to take a photo of the family fighting. Now future generations will only see the good times grandpa had. Is that a lie?
A People's History is littered with first hand accounts and journal entries. It is full of quotes to other authors who've collected similar. If you can point to the 95% of lies it contains, that would be great. I don't think you can though. I think the best you can say is that you disagree with the narrative, and to do so, you must ignore the real stories from real people.
These days, Howard Zinn, Chomsky, and most other left wing intellectuals of yesteryear have lost their cultural power.
It doesn’t help that most of these folks are 1. So sold into the “American intelligence bad” narrative that they look at every left wing crack point and act like nothing bad happened under them.
Even Allende, the classic example of a “good Marxist” leader, was a total asshole and likely deserved to be shot. Zinn isn’t going to tell you that or about when the US IC got it right… they also won’t mention that the ultra conservative death squads were still often the lesser of the two evils.
Sorry, but Howard Zinn is post modern neo Marxism. Might as well go read Deleuze or Foucault.
Also sad to realize that the left in general has been extremely neutered. The plot to “Canadian Bacon” by Micheal Moore is happening IRL and no one even knows that movie exists except random old American leftists.
Should people only read what's popular, what the dominant culture approves of? Are more unpopular ideas more wrong? You apparently dislike Chomsky; would you have liked Chomsky more when those books were more popular?
> the “American intelligence bad” narrative
You are ignorant of these issues if that is your understanding. Certainty also seems to support that, as does the ranting.
“Postmodern Neomarxism” basically doesn’t exist outside of a few fringe academics. Recommend you read Derrida’s _Spectres of Marx_, which goes into detail about how dead academic Marxism was even in 1993 (spoiler: it’s even deader now). His hopes for a “New International” never materialized and died with him.
Even Žižek’s more recent _Too Late to Awaken_ is a depressing slog of an elegy, and he is the closest thing to a real “Neomarxist” you could ask for.
Relatedly, Foucault and Deleuze do not write anything remotely like Zinn. Comparing them is wholly meaningless, they’re playing entirely different games.
That particular book is of special interest to me as a stirner shill/fanatic. The "Specter of Marx" he's refering to is Max Stirner! Derrida is trying to answer the question of "Why did Stirners book, The Unique and its Property, trigger Marx to write The German Ideology and dedicate over 300 pages to shitting on stirner?".
I've cover to cover read that book, and it's garbage dog shit just like Zizek is. The moment that you start trying to wax poetic about whose what ism is when you become post-modern neo marxism. It's academic ivy league eliteist careerism and its disgusting. Fashionable nonsense. Zinn and Zizek are playing the exact same game. One just writes more coherently than the other.
If you think Hegel or Marx are worth citing, you're a pomo/fraud. If you think Psychoanalysis/Freud are worth citing, you're a pomo/fraud. Deleuze killed himself. Mark fischer killed himself. Althussar killed his wife.
Wilhelm Reich was a loony who after writing his analysis of the sexuality of fascism went nuts about "Orgone" machines. Foucault, Sartre/De Bouvier were pedos and collaborators. Even my favorite half-breed of pluralism mixed with pomo shit, Arendt, was in love with the arch nemesis of "oh they were literally a nazi " pomos - Heidigger! And who can forgot Carl Schmitt?
JUST WRITE THESE IDIOTS ALL OFF ALREADY AND LEAVE THEM IN THE DUST BIN OF HISTORY!
Boring tech is boring because it is reliable. You don't need to fight constantly with it. You can forget about it, and focus on your work. Maybe reliable equals mature.
reply