My above comment wasn't aimed at you. Thanks for the response though! What you did is phenomenal, and I wasn't taking a dig at you in any way.
What I intended in my comment was that, may be starting from scratch (like you did many months before, and what PaulG suggested recently) is the way to go for future job prospects and startups. Most of the comments in the thread I posted was negative for PaulG saying that.
what you describe would be cheating. My approach is the opposite. What I did was "You are born during the exam, given access to a training set and the questions then learn from scratch during the exam"
1) weight update during eval: this is a form of test time training and not really cheating. It is also closer to Sutton's views of intelligence: models should learn during deployment, instead of being frozen after training
2) overfitting: agree that true measure is private set. It scores on private set roughly on par with TRM (a comparable model), obv with much lesser compute
--
also re LLMs: they do not follow this 3 split since (a) Incredibly hard to keep a pretrain dataset clean, (b) common in labs to benchmaxx during postraining (and known to do so on ARC)
> weight update during eval: this is a form of test time training and not really cheating.
Possibly "not really cheating", but it does make benchmark comparisons unfair - especially as the other models are unlikely to have their weights updated during the eval.
> In the university I first dropped out of, students that surpassed me studied by getting and sharing copies of previous exams and solving those question
Yes what you describe would be cheating. My approach is the opposite. What I did was "Carry your textbook to the exam and then learn from scratch during the exam"
What you described is cheating because more time than the exam permits. ARC was designed specifically to avoid this. The exam in question (kaggle competition) is 12hrs long with 4xL4s. I trained on 1.5hrs with a single 5090 (which converted to 4xL4s is slightly longer, but still within 12hrs).
(There's also access to experts who know the answer, which kaggle bans by banning the internet)
Your arguments btw support my work over the LLMs more. LLMs today are postrained with a large amount of synthetic ARC data. (Exactly the "teach to test" criticism). Thats why they perform so well on ARC. Base models still are terrible at ARC-2
I didn't phrase that well and cant edit, so clarifying:
What I did was "You are born during the exam, given access to a training set (which is curated and allowed) and the questions then learn everything from scratch during the exam"
>Yes what you describe would be cheating. My approach is the opposite. What I did was "Carry your textbook to the exam and then learn from scratch during the exam"
>What I did was "You are born during the exam, given access to a training set (which is curated and allowed) and the questions then learn everything from scratch during the exam"
This is getting into metaphorgotten territory, I'd have to do a deep dive to really understand whether it is "cheating" or, more formally, a methodology that encourages overfitting.
But for what is worth, taking the textbook to the exam (regardless of whether you were born there or not), would be cheating in a test as well. Although it is possible that in the model training context it does not lead to overfitting, it certainly doesn't preclude it.
>Yes what you describe would be cheating. My approach is the opposite. What I did was "Carry your textbook to the exam and then learn from scratch during the exam"
It is comforting that someone agrees, to this day this is considered a gray-area (or not even that) by students, graduates and those that look up to the University of Buenos Aires institution.
in non metaphor terms: In many ML situations you can carry the train set with you test time. Eg: KNN, SVM, replay buffers, etc
this is one such case
--
the separate overfitting concern is fair, look at private holdout performance for that. it performs on par with TRM (a comparable model) in the private set, ofc with far less compute
I think you can do whatever you want with your train set, what would be concerning is training with the test set.
I don't think overfitting is separate, the consequence of putting the benchmarks in your training set is not that you "cheat" by breaking some moral code, it's that it breaks the purpose of the benchmark and trains your model to be good at that benchmark only, instead of being generally useful.
there's a very large variance in doctors' abilities in India. At the very top they are close to the best in the world, esp with an insanely high workload.
but on an average, not great
Also, a lot of gymgoers and physical trainers I know hadn't heard of rhabdo either (and this is a relatively wealthy part of a tier 1 city)
reply