At SEMICON West, there always seems to be a theme. Last year, it was China. See my post SEMICON: China, China, China for what I wrote this time last yearr. In fact, I was so convinced that something big was happening that I went to SEMICON China too, in March this year. For my impressions there, see SEMICON China: Me and 70,000 of My Closest Friends . At the opening reception someone told me that actually there were 70,000 people expected at SEMICON China, but 90,000 showed up. By comparison, SEMICON West is 25,000 or so, about a third of the size. This year the theme is that we on the verge of a major transition. Different people called it different things, the 4th Industrial Revolution, the 4th Tectonic Shift, the AI era. But everyone sees the huge growth in silicon for AI as being a significant change. Some choice quotes: "embrace the oportunity, I don't think you will see it again in your lifetime" "software is eating the world...and AI is eating software" "hardware is sexy again" "automotive will be more wafer-starts-per-month than mobile" "semiconductor will be $500B by 2020" "semiconductor is going to be over $450B this year based on this weeks new forecasts" Melissa Schilling The openig keynote was by Mielissa Schilling of NYU. She has been doing research on serial entrepreneurs, what makes them tick. It started when she wondered how Apple would be after Jobs: was it specifically him or was it something more. She started a multi-year project to find and analyze not innovators, but serial innovators, people who had done breakthrough work again and again throughout their life. She ended up studying Albert Einstein, Steve Jobs, Elon Musc, Nikolas Tasla, Benamin Franklin, Thomas Edison, Marie Curie, and Dean Kamen. She has a book out about this, called Quirky: Remarkable Stories of Breakthrough innovators . I won't go through everything that she talked about, but she discovered 4 traits that seem to be common to all these people. Some are surprising. Sense of separateness. Every single one (except Benjamin Franklin) had a strong sense of separateness and that they didn’t belong to the social world. For example, Einstein had a hard time in academia. Every physics department in Europe turned him down. So he got a job as a patent examiner. But as a non-academic, he didn’t do that academic stuff and cite everything that had gone before. He just cast aside all these concepts from Newton and developed revolutionary ideas in physics that broke with tradition. Now he is acknowledged as a genius but academia was actually very slow to accept his ideas. Pierre and Marie Curie gave their kids to his father to raise so that they could focus on science. Self-efficacy, the sense that you can overcome obstacles and succeed. For example, when Elon Musk said he was going to build reusable rockets, everyone who had been involved in space before said "Do you think we never thought of that? It won't work." But Musk was confident, "I think it will work." And it did. Seven of the eight (Edison being the exception) had an idealistic goal that mattered more than leisure, money and sometimes even families or health. Being in the right place at the right time. None of these innovators started with much money, even if they were rich later in life. Elon Musk ran away from home to the US when he was 17 because his father would only pay for him to go to university in Pretoria. More important than access to money is access to other people, such as being in silicon valley at the start of the semiconductor and PC revolution. That in itself is a weird series of coincidences, such as Shockley's mother getting sick. For my take on that see Who Put the Silicon in Silicon Valley? John Kelly The next keynote was right on topic, with John Kelly of IBM talking about The Era of Artificial Intelligence . He started by saying: We have never been at a more exciting point than we are today. Two things are happening that are going to change not just our industry but the world. He had very big eras so we are just in the third in his taonomy. The first was the tabulating era from 1900s to 1940s. That was followed by the programmable era from the 1950s to today. The era of the type of computing we all grew up with, where we took human processes and knowledge and told the machine what to do. Underlying it was Moore's Law so that we have got a long way from when John started and was trying to cram 1000 transistors onto a chip. Now we are entering the cognitive era, starting a few years ago. This is computing at a scale that will dwarf the programmable era in ways that will change all of our businesses, and our lives. It becain around 2011 (Watson) and it has exploded. This will be 50 60 or more years There are three exponentials too. The first is Moore's Law. The second is Metcalfe's Law, about the growth in network traffic and the value of a network being realated to the square of the number of nodes. The third exponential is upon us, which John called "Watson's Law" after IBM's Watson AI system. Data is doubling every year or two and there aren't enough programmers to deal with the data and extract value. The only way forward is machine learning and AI. A big question that came up in several presentations during SEMICON was "why now?". John's answer was that there are three drivers. One, data is exploding. Two, the cloud gives us massive scalability for comput workloads. And thirdly, there are a lot of advances in neural networks, machine learning, and deep learning. AI = data + algorithms + compute. Putting all that together has resulted in the world's most powerful computer, the Summit, at 200 petaflops. It is also the largest AI system in the world, between the IBM Power processors and the NVIDIA GPUs. The problem, and the challenge for all of us, is that it takes 13MW of power. That is all in a room about the size of the auditorium SEMICON keynotes were being held in, the theater at Yerba Buena Center for the Arts. Also, as it happens, the stage where the iPhone was revealed by Steve Jobs in 2007. The next supercomputer will be 5-6 times bigger still in power. Why is this important? "If you want to see the future of computing, then look at supercomputing." Today's consumer product is yesterday's supercomputer. John reckons you will never see another non-AI supercomputer. "That is the roadmap for computing." Except that there is a fourth exponential where IBM is working, namely quantum computing. IBM have announced 50 qubits in the lab, and also 20 qubits are available on the net for use by anyone. Once the number of qubits gets up to about 100 then it will be possible to do calculations in seconds that the largest computers in the world would not be able to do at all. Moore's Law for 50 years will be exceeded with the first commercial step of a quantum computer. The dream is to put AI on a quantum computer and then go after the world's hardest problems. Bill Dally The last presentation I'll cover in this post is Bill Dally, who is the chief scientist at NVIDIA. He is famous for suggesting ages ago that with the end of Moore's Law looming, parallel computing would be the way forward. Now that seems obvious but it was controversial at the time. The end of Moore's Law means we don't get more performance for free: It is an exciting time to be a computer architect since now we have to start getting clever to deliver the value. Bill had his version of the "why now?" question, given that all the basic algorithms have been around since the 1980s. It needed large labeled datasets was one missing ingredient. But mainly it was hardware powerful enough to run large models on large datasets in a reasonable period of time. Progress is now being gated by hardware. The multiplicative effect of data and model size is leading to a quadratic increase in the time needed to train. To put in perspective, ResNet-50 takes 7.72 billion operations to process one 225x225 pixel image 230Gops for 30 fps video 9.4Tops for HD for one camera 10-12 times that for a car since there are about a dozen cameras The Volta V100 has 125 Tensor TFLOPS (half-precision core inner loop ofr deep learning). There are 32 GB of HBM2 (those are the 4 black die you can see around the GPU chip iteslf). It does matrix multiply as a single instruction so there is very little overhead from instruction processing. The big message here is "that you don't have to give up programmability to get specialization." NVIDIA have open-sourced their deep learning accelerator software stack. They are pursuing robotics, image processing, self-driving cars, but for other applications they want other people to have access to the technology. They have also been dong some aggressive work on cutting down the size of neural networks. First, by reducing precision. This saves a huge amount of power since an 8 bit multiply is only 1/16th the power of a 32-bit multipley. The datastore gets smaller too, so more is close to the processor. Each level of memory hierarchy is another order of magnitude in power and loss of performance. The network can also be pruned, taking out neurons that are not "pulling their weight" and then retrain. It turns out you can throw away 90% of them and still have the same accuracy (70% for the convolutional layers). So Bill's conclusion is that the future is dep learing, enabled by GPUs and fixed function accelerators. Infreases in efficiency can be achieved with reduced precision, pruning, and efficient networks. Sign up for Sunday Brunch, the weekly Breakfast Bytes email.![]()
↧