В аду должно быть отдельное место для людей, которые постят ссылки на arxiv сразу на pdf. 👹

Сначала пошли результаты без методов (GPT-3+, Gemini и т.п.), теперь идут методы без результатов (https://arxiv.org/abs/2404.07221).

Препринты превращаются в недопринты.

Хорошее дополнение к "Библии промпт-инженера" — Библия уровнем выше, паттерны проектирования агентов :)

Agent Design Pattern Catalogue: A Collection of Architectural Patterns for Foundation Model based Agents
https://arxiv.org/abs/2405.10467

Красивая работа, не увидел её раньше.

"Meanwhile, we recommend summarizing the recent trend of compute growth for notable and frontier models with the 4-5x/year figure. This should also be used as a baseline for expectations of growth in the future, before taking into account additional considerations such as possible bottlenecks or speed-ups.

Compute is the best predictor of broad AI capabilities we have, and so tracking its growth is fundamental to forecasting the trajectory of AI. This piece provides an updated view of this crucial topic and argues for a growth trend that has not slowed down in recent years."

https://epochai.org/blog/training-compute-of-frontier-ai-models-grows-by-4-5x-per-year

По ссылке интерактивный документ с живыми графиками, можно покликать и посмотреть данные конкретных моделей. Данные с 2010 по май 2024.

Планируйте, что через четыре-пять лет топовые модели будут обучаться на 1000x более производительных системах.

Там же интересная работа про стоимость обучения моделей:

"Our analysis reveals that the amortized hardware and energy cost for the final training run of frontier models has grown rapidly, at a rate of 2.4x per year since 2016 (95% CI: 2.0x to 3.1x). We also estimated a cost breakdown to develop key frontier models such as GPT-4 and Gemini Ultra, including R&D staff costs and compute for experiments. We found that most of the development cost is for the hardware at 47–67%, but R&D staff costs are substantial at 29–49%, with the remaining 2–6% going to energy consumption.

If the trend of growing training costs continues, the largest training runs will cost more than a billion dollars by 2027, suggesting that frontier AI model training will be too expensive for all but the most well-funded organizations."

https://epochai.org/blog/how-much-does-it-cost-to-train-frontier-ai-models

Я недооценивал косты на R&D staff.

Продолжение истории

https://ssi.inc/

Safe Superintelligence Inc.

Superintelligence is within reach.

Building safe superintelligence (SSI) is the most important technical problem of our time.

We have started the world’s first straight-shot SSI lab, with one goal and one product: a safe superintelligence.

It’s called Safe Superintelligence Inc.

SSI is our mission, our name, and our entire product roadmap, because it is our sole focus. Our team, investors, and business model are all aligned to achieve SSI.

We approach safety and capabilities in tandem, as technical problems to be solved through revolutionary engineering and scientific breakthroughs. We plan to advance capabilities as fast as possible while making sure our safety always remains ahead.

This way, we can scale in peace.

Our singular focus means no distraction by management overhead or product cycles, and our business model means safety, security, and progress are all insulated from short-term commercial pressures.

We are an American company with offices in Palo Alto and Tel Aviv, where we have deep roots and the ability to recruit top technical talent.

We are assembling a lean, cracked team of the world’s best engineers and researchers dedicated to focusing on SSI and nothing else.

If that’s you, we offer an opportunity to do your life’s work and help solve the most important technical challenge of our age.

Now is the time. Join us.

Ilya Sutskever, Daniel Gross, Daniel Levy

June 19, 2024

Антропик выкатил новую модель Claude 3.5 Sonnet

https://www.anthropic.com/news/claude-3-5-sonnet