☆ Yσɠƚԋσʂ ☆

  • 6.81K Posts
  • 4.66K Comments
Joined 7 年前
cake
Cake day: 2020年1月18日

help-circle



  • Instead of shrinking components, manufacturers have been focused on developing new hardware that is tailored to AI. This has provided occasional incremental improvements in performance, but none has come close to keeping up with the exponential curve of AI’s increasing demands. Ultimately, inefficiency may be of little concern to the people within the tech industry who believe that they are replicating intelligence itself. There is an almost-religious conviction among many in Silicon Valley that something mindlike could arise from LLMs, which are ultimately just statistical language-generating software—this, despite the software’s inability to recall basic facts, its lack of common sense, and its complete dissimilarity to a biological brain. Even Yann LeCun, one of AI’s “godfathers,” told The New York Times recently that “LLMs are not a path to superintelligence or even human-level intelligence.” But the mythological lure of AI is so strong that many engineers believe that nothing should stand in their way. Not even the basic task of writing efficient software.


  • Seems like might be problem on your end, perhaps your ISP is blocking it?

    As they scramble to keep their systems online, AI companies are making things expensive for the rest of us. Large language models such as ChatGPT and Claude are so resource-hungry that tech companies may be purchasing 70 percent of the world’s supply of high-end computer memory, causing a shortage. As a result, the prices of computer memory and storage are skyrocketing: Hard drives that I bought for my reporting two years ago for $350 each were $800 when I checked two weeks ago, and are now out of stock. The prices of some laptops have gone up as much as 50 percent, and low-cost computers are being hit the hardest. Affordable entry-level computers may “disappear by 2028” according to one forecast. And the memory shortage is expected to continue for years. The memory is being put into data centers, which tech firms are expanding at incredible speed. They are planning to multiply total U.S.-data-center capacity by a factor of eight over the next few years. The demand for electricity at these sites is already so great that some companies are repurposing jet engines to power them. The problem is not simply that AI is being deployed so widely or quickly. Other computer technologies have seen similarly massive growth without triggering such a large spike in electricity or a shortage of computer components: Video and music are now streamed around the globe, accounting for many terabytes of internet traffic daily; the smartphone boom required the manufacturing of billions of devices that are now transferring huge amounts of data; billions of household devices are also now part of the Internet of Things; and whole industries have moved their operations to cloud software, which is hosted not in the sky but in, yes, data centers. Read: The $10,000 MacBook Pro is here The problem with generative AI, in the industry’s own jargon, is that it does not scale. The cost of growing from, say, a thousand users to a million is a key factor that venture capitalists examine when they evaluate start-ups. They want to see that the cost of adding each new user decreases over time, so that the company can support millions of users and make increasing profits. This is achieved partly through the careful engineering of computer systems that can efficiently handle more users who want to post photos, hail Ubers, or stream music. With generative AI, the work of building efficient, scalable systems has not been done. And the problem is exacerbated by the ever-larger generative-AI models, which have grown from 175 billion parameters in 2020 to more than 1 trillion today, according to independent estimates (the actual sizes of the models powering products such as Claude and ChatGPT are secret). The large in large language model should not be a selling point. But the industry’s observation that bigger models tend to outperform smaller ones has given rise to a totemic belief in “scaling laws” that suggest any problem can be solved by simply making models bigger. “Maybe with 10 gigawatts of compute, AI can figure out how to cure cancer,” OpenAI CEO Sam Altman wrote on his blog in September. Recommended Reading

    A photograph of children on and around a dining-room table, making a mess
    The Myth That Gets Men Out of Doing Chores
    Joe Pinsker
    Bottom half of woman's face with a reflection in a mirror
    How Not to Be Your Own Worst Enemy
    Becca Rashid and Arthur C. Brooks
    An illustration of a husband and wife sitting on opposite sides of a bed that's floating in open water.
    Dear Therapist: Can I Still Trust My Husband?
    Lori Gottlieb
    

    Yet the returns are diminishing. The bigger an AI model is, the less it improves with each added parameter, and so it must be made bigger at a faster rate just to sustain steady progress. I asked a few AI researchers whether they could name any other real-world software that scales so poorly. None of them could think of any. Even outside the world of software, it’s hard to find a comparable example, given that economy of scale is the principle that has made light bulbs, cars, and clothing so affordable. By economic and engineering measures, generative AI might be the worst technology ever deployed. Read: Welcome to a multidimensional economic disaster But with the massive investment behind the current bloated approach, there may not be much will to change. Ilya Sutskever, a co-founder and former chief scientist at OpenAI, said in a November interview that companies take the brute-force approach “because it gives you a very low-risk way of investing your resources.” It’s harder, he argued, to invest in research that would reengineer a product currently accruing trillion-dollar valuations. Those who suspect we are in an AI-driven bubble economy have pointed out that the profitability of these companies remains an open question, largely because of the high cost and inefficiency of the technology. Efficiency is a core principle of computer science. One of the first things undergraduates learn is that writing a program that sorts a list of 50 words is easy. But if you give that program 50 million words, it will likely run out of memory or take hours to finish. Much of computer science is learning the clever coding techniques that prevent this from happening. Many of these techniques take advantage of repeating patterns in the data so that as the program receives more input, it takes less time or memory to process each additional bit. Such efficiency is one reason that modern smartphones and computers are so capable and affordable. This is called logarithmic scaling, and it looks like this when you graph it: graph showing what tech companies want to see when they scale Large language models do not scale logarithmically. When they’re given more words to process, they get slower and use more memory—the time and resources increase faster as the input grows. In technical terms, LLMs scale quadratically. Any computer-science student knows that this is very bad. Epoch AI, an organization that tries to determine the costs of operating AI models, published a graph last year, which is reproduced here with permission. It shows the exponentially increasing costs of serving more “tokens”—the words users type to chatbots—with several public AI models. graph showing what AI companies are actually seeing as they scale LLMs AI does not have to be built this way. Traditionally, the goal of AI was to solve problems in ways that simulated human mental processes. Researchers observed their own thinking and tried to implement their mental habits in code. This approach has mostly been abandoned, partly due to the difficulty of discerning and articulating the rules of human thought, but it did have the benefit of consuming far fewer resources and data. Today’s approach to AI doesn’t try to describe the rules of human thought; instead, it gives a computer millions of examples to imitate. The huge quantity of examples is one reason that large models can perform better than small ones when generating language, images, and music—they have more material to draw from. Some researchers want to bring back the old, more efficient approach and combine it with the modern approach, but so far these projects have not drawn nearly as much attention or funding as the models that power chatbots. Chatbot companies are aware that their products are inefficient. Some have found techniques for improving performance, but they have not yielded significant gains. Occasionally, companies claim to have made breakthroughs—Anthropic CEO Dario Amodei has called them “compute multipliers”—but they are usually described in vague terms, and there is no evidence that the basic problems of quadratic scaling and exploding model size have been overcome. (Anthropic declined to comment on the record when I reached out to ask about this.) Some researchers are working on extremely small models that require fewer data and less computing power. I spoke with Alexia Jolicoeur-Martineau, an AI researcher at Microsoft who independently designed one of these, and asked her about the industry’s brute-force approach. “It’s a bit insane,” she told me. “At some point you have to learn to be a bit more efficient.” Last year, Jolicoeur-Martineau won a $50,000 prize for her paper on a “tiny recursive model” that doesn’t consume huge quantities of computing resources. “The idea that one must rely on massive foundational models trained for millions of dollars by some big corporation in order to achieve success on hard tasks is a trap,” she wrote. Her model isn’t a substitute for an LLM—it’s designed to solve logic problems in fields such as biology and electrical engineering, rather than generate language—but it can do some of the tasks that much larger AI models are currently being used for. Yet we seem stuck with LLMs, perhaps because they have been so aggressively marketed. They are now being added to everything, whether you want them or not. In 2024 and 2025, they were integrated into both Windows and MacOS, which means that more computing power is now needed to run a basic personal computer. Smartphones are also being sold with upgraded hardware as companies anticipate new AI features. Inefficient AI is also being added to common programs such as Adobe Photoshop and Microsoft Word, meaning that computers need to be more powerful to run this software. Ted Chiang: No, artificial intelligence is not conscious This is all especially bad because computers are no longer improving at the rate they used to. Since the 1950s, manufacturers have learned to make microchips steadily faster, smaller, and cheaper, a trend known colloquially as Moore’s Law. But in the past few years, components have gotten so small that manufacturers have run into molecular-level limitations on shrinking them any further, which has slowed progress significantly.


  • I think the big picture here is that the difference in quality is largely subjective at this point, while US companies are burning through orders of magnitude of cash which is obviously not sustainable. We shouldn’t underestimate the power of developing things in the open. Chinese open models benefit from the wisdom of an entire global research community while American engineers working on proprietary closed models are working in their own insular silos. It should be no surprise that the scientific community at large would pull ahead of these small teams. On top of that, doing research in the open amortizes the cost. Incidentally, this is exactly the same logic that led open source to dominate in recent years.


  • The energy consumption trope in particular hasn’t been true for a while now. You can literally run models on your laptop for most tasks, and these are models that have capability that needed a data centre literally less than a year go. Meanwhile, the whole notion of people offloading their cognitive capacity to models is based on a handful of studies with tiny samples. So yes, you are wrong, and you just run around uncritically repeating nonsense thinking you’re being really profound.





  • Incidentally, Asimov wrote a great review of 1984 basically calling Orwell an emotionally stunted manchild who wrote a tantrum disguised as a novel. Orwell was a guilt ridden rich kid who played hippie in the slums then got his feelings hurt in Spain and spent the rest of his life writing revenge fantasies against Stalin instead of actually trying to understand how the world works.

    1984 was just boring and lifeless rant where nothing happens, and the technology is nonsensical like having humans watch every citizen on camera instead of just using computers which even Asimov in the 1980s could see was laughably inefficient. Orwell couldn’t imagine women as anything but brainless sex objects or proles as anything but subhuman animals and he thought ballpoint pens were worse than actual scratching steel nibs. The guy was a technophobic elitist with zero forward vision who projected his personal grudge onto the future and millions of people treat his little hissy fit like prophecy. It’s frankly incredible that his shallow writing became a cultural icon in the west.

    https://www.newworker.org/ncptrory/1984.htm











  • Oh yeah forgot about limewire. :) And I’ve been noticing the same trend, a lot of people don’t really know how to torrent things anymore. It really is a strictly superior user experience. Once you have a file then nobody can take it away from you, it doesn’t expire, you can play it on whatever device you want whenever you want. The fact that people accept streaming along with all the limitations is really crazy to me. Even outside the question of cost, it’s just a shittier overall experience.


  • I am talking about interrogating the implications of the present world we live in. What I was pointing out is that the author makes a shallow analysis of the symptoms without following the threads to identify the root causes. You don’t need to be a philosopher to do that.

    I’m also not talking about an alternative history where AI might’ve emerged in a different form. I was pointing out the underlying causes of the negative effects associated with how this technology is used, and we need to be clear on that in order to do anything about the problem. The issue is capitalist control, and the solution is to develop this technology under public ownership the same way other open source technology is developed. Open alternatives from China are already the biggest threat to the whole model, so this is already starting to happen.

    I think people who are opposed to the way this tech is used should be thinking of how to wrestle it away from corporations, and to build it in the open. This is the whole concept behind having ownership of the means of production. In my opinion, that’s the only realistic solution to the problem in the long term.








  • The analysis in the post remains firmly at the level of phenomena, failing to address the fundamental contradiction of the capitalist mode of production. The blog post correctly points out that tools are not neutral since they shape people and social relations. All tools are material manifestations of specific production relations. However, the blog post treats AI as a tool or technology itself, failing to question why AI exists in this specific form at the current historical stage. The reason there is a tendency towards large scale models which are centralized and commercialized is that they are an inevitable product of the logic of capitalist accumulation.

    From the perspective of the base and the superstructure, after the highly developed capitalist productive forces, capital urgently needs a new means to accelerate circulation, reduce labor costs, and open up new areas of accumulation. So, the inefficiency and high energy consumption of AI are not technological defects, but rather a price that capital is forced to pay under specific historical conditions because true efficiency in form of distributed, open-source, and democratized AI cannot serve the maximization of monopoly profits. The waste, environmental damage, and ethical crisis of the AI ​​industry are essentially inherent contradictions of the capitalist mode of production. The problem stems directly from the contradiction between social production and private ownership.

    The post also discusses the idea of tools shaping people, but fails to clearly distinguish the different shaping forces of use value and exchange value on technological development. Under capitalism, AI primarily serves the needs of capital accumulation rather than the comprehensive development of a society. The whole AI makes humans stop thinking and stop creating argument is just describing the deepening of capitalist labor alienation in the digital age where workers are alienated from creative labour. The push to replace rather than enhance human capabilities is driven by the need for replaceable, standardized labor, rather than independent thinking subjects.

    Merely demanding rational use or ethical norms without addressing the private ownership of the means of production can only alleviate symptoms while doing nothing to address the root cause. The laws and ethics of capitalist society are themselves part of the superstructure and their fundamental function is to safeguard the interests of the bourgeoisie. As long as the means of production remain in the hands of a few monopolistic capitalists, any calls for ethical use are just empty moralizing.

    Finally, it’s worth noting that there is little room for letting us do what machines cannot do like appreciating predecessors and fighting for policy within the capitalist framework because such behaviors are systematically marginalized. True liberation lies in breaking down the social relations that determine the direction of technological development. The goal has to be to move the development of AI from serving capital accumulation to serving the free and comprehensive development of humanity.

    The core problem with the critique in the blog post is that while it is emotionally charged, it fails to rise to the level of a systematic analysis of the capitalist mode of production. Merely calling for critical use or humanistic concern is insufficient because it is essential to understand the relations of production in order to see past the illusion of tools controlling people.