"With GPT-3 costing around $4.6 million in compute, than would put a price of $8.6 billion for the compute to train "GPT-4".
If making bigger models was so easy with parameter partitioning from a memory point of view then this seems like the hardest challenge, but you do need to solve the memory issue to actually get it to load at all.
However, if you're lucky you can get 3-6x compute increase from Nvidia A100s over V100s, https://developer.nvidia.com/blog/nvidia-ampere-architecture-in-depth/
But even a 6x compute gain would still put the cost at $1.4 billion.
Nvidia only reported $1.15 billion in revenue from "Data Center" in 2020 Q1, so just to train "GPT-4" you would pretty much need the entire world's supply of graphic cards for 1 quarter (3 months), at least on that order of magnitude."

Looks like where will be no significantly larger model in GPT4 for us :C
source - https://www.reddit.com/r/MachineLearning/comments/i49jf8/d_biggest_roadblock_in_making_gpt4_a_20_trillion/

just have found an amazing site - https://code.likeagirl.io/
really awesome concept that may inspire someone

github finally stopped looking like a thermonuclear explosion at night

GET ON THE HYPE TRAIN

Why there's no email client which allows to read mail on the one side and to answer in in the vertical panel nearby? I have to do THIS, and it's disgusting.

I have a good news and a bad news.
Good news: original mass effect trilogy will be continued.
Bad news: original mass effect trilogy will be continued.

https://youtu.be/Lg-Ctg6k_Ao

«You must respect other people's opinion»
Other people's opinion: