Futapy

HW 8

per http://informationcomputingandthefuture.blogspot.com/p/hw8.html

A1:

I shall choose Radio as my invention. I should like it to be clearer (improve Loss of Information), cheaper ... no wait. I shall choose the Electric Car Battery. I should like it to be easier to repair (34), more powerful (21), and weigh less (1), to pull three easy improvements out of a hat. I'd like it to be cheaper and last longer also, but I'll stick to the first three.

Ease of Repair conflicts with Weight of moving object [2, 27, 35, 11] and Power[15, 10, 32, 2], and Weight of moving object conflicts with power [12, 36, 18, 31] and ease of repair [2, 27, 28, 11] (I find it odd that 1/24 conflict list is different than 24/1 conflict list).

Right now EV batteries are difficult to repair for a number of reasons, but the biggest being that they are confined in a monolithic pack which is the heaviest single component of the vehicle, and typically located low under the vehicle to make the vehicle handle better. To make repair easier, segmentation could be used, but that would add more weight to the structures holding smaller battery segments.

A more powerful battery could be made larger and denser and even more difficult to repair. Or a less weighty battery would certainly be less powerful. Of course weight impacts power in that heavier objects take more energy to get moving, and a more massive battery conflicts directly with ease of repair. I think these are really the same issue so I'll focus on Repair / Weight conflict and Weight of Moving Object / Power conflict.

Using this nifty TRIZ solution map, Ease of Repair / Weight conflict suggestions:

Weight of Moving Object / Power conflict

This wasn't easy. Perhaps I shouldn't have chosen such a highly engineered component to try to improve.

Part B:

How about light bulbs; for many decades they pretty much made yellow light by heating a piece of metal, then fluorescent bulbs gave us a way to make different colors (think Vegas), then LED lighting gave us on-demand color changing by remote control (have you seen the North Forest Lights exhibit up at Crystal Bridges? It's a pretty cool use of coordinated color and light texture changes); even some light bulbs sport a wireless bluetooth speaker to play music or set the mood. What might come next for the humble light bulb? How about AI driven "smart lights" that would monitor the room and provide just the right amount of illumination for any task, and then turn off when that task is done. How about lights that are networked with sensors on the electric grid to dim to only a minimum fraction of their possible light when there are shortages of electricity, in return for more favorable electric rates. Or maybe lights that consist of a phased array of coherent light (laser) emitters that could cheaply focus and project patterns and information on any surface? I think the future is bright for light bulbs.

A2:

Add the meat of the paper - if I am going to say anything about anything, it will be this: What is GPT3, what is OpenAI actually doing vs. Google, and how close is GAI to becoming what we would think of as human equivalent?

AI capability has made remarkable progress in the last few years, not through breakthroughs in research but rather massive funding and scale-out of existing techniques. The results are astounding and will almost certainly change our lives. Sam Altman states "The changes coming are unstoppable. If we embrace them and plan for them, we can use them to create a much fairer, happier, and more prosperous society. The future can be almost unimaginably great." (https://moores.samaltman.com/)

OpenAI is an artificial intelligence research company founded with a goal to "advance digital intelligence in the way that is most likely to benefit humanity as a whole". OpenAI was founded by Elon Musk and Sam Altman as a non-profit, but with total funding commitments amounting to $1 billion USD, allowing them to make a bet on the scaling hypothesis and develop and train relatively simple models at a massively parallel scale and achieve generalized performance equal to the best purpose-built and fine-tuned models.

GPT-3 is (what exactly?): From OpenAI's GPT-3 paper, GPT-3 is "an autoregressive language model with 175 billion parameters" (todo: explain parameters) trained on a massive amount of text from the Internet using a high-bandwidth cluster of GPUs provided by Microsoft. Using this approach GPT-3 performs natural language processing (NLP) tasks similarly to state-of-the-art fine-tuned models. Given only a few examples of a task, called "few shot training", this model performs as well as other models that have to be manually fined tuned to the problem at hand with many thousands of examples. Moreover, the GPT-3 language model performs well on a wide range of general language tasks "including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic". Due to concerns over misuse of the model, OpenAI has chosen not to release the trained model, estimated to fit on a 1TB hard drive, over concerns of misuse including the automated generation of "misinformation, spam, phishing, abuse of legal and governmental processes, fraudulent academic essay writing and social engineering pretexting."

In 2018, OpenAI published a blog post on the scale of the "Compute", shorthand for computing power on graphics processing units (GPUs) used to train various other models, and noted that from the 1960s through about 2012 the time for the Compute to double was about 2 years, roughly in the same rage as Moore's law. However since 2012 they note the trend is a doubling time of closer to 3.4 months. At the time of their post (May 2018) the largest models were trained using about a thousand petaflop/second-days (pfs-days), or "performing 1015 neural net operations per second for one day, or a total of about 1020 operations"); extrapolating on the same trend line to April 2021 indicates that as many as 10 doublings in the computational power used to train leading models may have occurred since, or models being trained today with the computational equivalent of a million pfs-days, roughly 1026 operations. In June 2020, Lambda Labs reported that GPT3 with 175 billion parameters "required 3.14E23 FLOPS of computing for training", which would reportedly cost over $4.6M at a cloud service provider.

The Bitter Lesson, an essay by Rich Sutton, deserves a mention here. The notion is that for decades AI researches have been building programs based on their understanding of the human brain, and these constructs perform well in the short term but are inevitably surpassed in performance by methods that don't include them but do use exponentially more computing power in a brute-force method. Gwern writes on the Scaling Hypothesis that weak neural networks scaled up outperform more powerful neural networks.

Not part of the assignment but a note to myself and the professor: Okay I have to admit, I am having real trouble here. I am in the valley of despair; I know nothing. I am spending a lot of time trying to understand a field I have had little exposure to in order to try to understand something like the history of AI and what GPT3 really is. It is taking hours to write just a few sentences. I fear I have bitten off way more than I can chew. I get terribly frustrated when I find well documented pages like this that are essentially what I wanted to produce or this which is unfathomably more complex and I think the amount of time needed to get from here to anything worthwhile is insurmountable. While I hope to be able to get caught up a bit on the project in the next week, I'd like your advice on how to craft this term project into something acceptable which I will be able to complete in the time given (and it has been getting the short end of the stick lately between mountains of day-job work, family demands among covid-death fallout, lecture, book and regular homework). I feel like I'll be doing good if I can understand and summarize this reddit post, much less write a coherent paper.


A3:

Predictions, pages 96 - 117: essays on and "predictions" by Umberto Eco and Francis Fukuyama. I have predictions in quotes because Eco tries not to make a prediction; the sentence "perhaps completely new facts and insights will emerge; some of the divisions of knowledge may disappear" comes closest, when talking about understanding the mystery of human consciousness (a theme mentioned by several of the previous voices in the book). The short biographical essay on Eco written by Domenico Pacitti had me reaching for the thesaurus, and introduced the study of semiotics, commonly defined as the theory of sign systems in language, of which Eco was a professor. Eco's book Kant and the Platypus addresses the problem of how we assign meaning and how we perceive things through a series of fictional philosophical essays on how the German philosopher Immanuel Kant would formed an idea of the identity of a platypus (an animal seemingly assembled from spare parts from other animals); from this description I am very tempted to ... scratch that ... I have ordered from abebooks.com for $4.04 including shipping (cheaper than a trip to the library). He concludes that words acquire meaning through negotiation within a particular community of speakers. I somewhat disagree; in my opinion negotiation requires two parties aware of each other, and it is clear that an AI can assemble meaning from vast training data without negotiating with anyone. That may be a naive view; I'll find out after reading the book. He also puts forth the notion of limits in the interpretation of text that are bounded by empirical facts and common sense. He believes in common sense, even stating that it is the basic tool in philosophical thinking. In the last few years particularly, I am less convinced that common sense is common, or that empirical facts are accepted universally as truth. Still, I look forward to receiving my new $4 book. If I ever have time to read it is another matter entirely.

Francis Fukuyama, at the time of this writing (1999), is an optimist. He states that the march of political development is progressive, the development of morality is cyclical, and he has high hopes for the coming century in light of the events of the late 20th century. He hits on a theme that I agree with: our most basic drives and instincts lead us to create rules that bind us together into communities, and it is in our genes to invent moral rules that constrain individual choice - this seems similar to the macro- and micro-evolution feedback cycle I discussed in a previous blog post. However he extends his philosophy "with reactionary views about the nuclear family, and his insistence that capitalism is not only successful but right", which might have had more traction in 1999 than in 2021. His publication The End of History and the Last Man is very bullish on western liberal democratic culture and states that capitalism the best economic policy for exploiting technology (how is this different from the neo-liberalism that is failing in Chile due to exploitation of the working class?). I am troubled by authors that assert the superiority of their ideas. His prediction is that "the further integration of global markets will further enforce the norms and institutions of the liberal democratic West". He discusses demographics and fertility rates, and predicts that English speaking countries will do better than France, Germany or Japan because the former have a better history of "accepting immigrants and assimilating them into a broader mainstream culture"; this might have been evident in 1999 but in 2021 it is clear there is a growing backlash against this assimilation in Britain and the United States that not only limits progress in the direction of liberal democracy but decreases the level of trust inherent in our societies, which Fukuyama himself sees as a bellwether of a nation's well-being and ability to compete. In all an interesting read but definitely written before the events of 2001 at the World Trade Center and the wars, fears and suspicions sparked from that event. Fukuyama is still around today and deserves a follow on twitter; it will be interesting to see if and how his philosophical tone has changed over the last two decades.

Other than the slightly difficult writing of essayist Domenico Pacitti that left me parsing words and asking my live-in English major the meaning of some words and phrases, my opinions of the book itself relative to other reviewers have not changed with this reading.