AIs and Starry Skies

Searching for insights about artificial intelligence in the night sky

This edition of Cannabis Uncaged is going to take a diversion from cannabis into the frontiers of AI. I hope you enjoy it. Apologies if you don’t. I’ll get back to cannabis next time.

This post is about how the patterns you perceive in a starry night sky may help explain the emergence of “human level” intelligence in computers.

Planted Seeds

My fascination with AI began when I read Nick Bostrom’s SuperIntelligence in 2015. After that book, I began by reading everything I could find on the subject, using Mr. Bostrom’s cited works as a starting point (within the bounds of what I could understand - much of the math and computer science is above my grade level).

My interest in AI ranges from “is this possible?” to “how is this possible?” and then definitely into “what does this mean for my life and work?” and absolutely into “what does this mean for humanity and the nature of the universe?”

AI is my favorite kind of subject: The more you learn, the more you realize how much there is to learn - until it becomes clear you could never learn it all. A prime example of power of the human colossus.

A Role For Purposeful Consumption

Wait. I can make this post about cannabis, too. Here it is: I suggest getting high and reading some books on AI. Cannabis can help you begin to make sense of the incomprehensible. Help you see things from a different perspective. For me, it’s what I find beyond the high that solidifies my love for cannabis.

Disclaimer: I’m going to do my best with this subject. But I wouldn't consider myself an expert in graph theory or machine learning or astronomy. If I get facts wrong, miss a logical step or two, or draw any premature or incorrect conclusions — please correct me.

As always, I reserve the right to update my opinions when presented with new information.

Parrot or Person

In artificial intelligence, especially within the context of language models, the world is venturing into uncharted territories where the lines are blurring between computational processes and the essence of human-like understanding. 

The crux hinges on a deceptively simple and profoundly complex question: Do these AI models "understand" the text they generate, or are they merely stochastic parrots, echoing back an amalgamation of their training without a grasp of meaning?

A recent exploration into this domain, described in an article from Quanta Magazine (which does a much better job of explaining it than I could ever do), has sparked a fascinating kerfuffle in my head and my goal in this post is to try to get that kerfuffle out in words. 

The new theory about Large Language Models (LLMs) described in that article posits that as LLMs grow in scale, trained on ever-expanding datasets, they begin to exhibit new, emergent abilities that suggest a form of understanding — one not explicitly present in their training data.

This debate may at times feel a bit academic (in the “not of practical relevance” sense), but I assure you it’s not. It hits at the heart of what we perceive as meaning and intelligence, both artificial and organic, and challenges our preconceptions about the nature of understanding itself. 

Yeah. Take another puff.

Random Graphs

Enter “random graphs.” Random graphs, as I fondly recall from my academic adventures, offer a framework for understanding how patterns can emerge from chaos. 

I’ve forgotten many of the things I’ve learned about math, but random graphs have stuck with me - and I know exactly why - it’s because they offer insights into understanding the patterns we perceive in the night sky.

In this next section, I’m going to attempt to describe a few key things about Random Graph Theory as they pertain to explaining our perception of the night sky to set us up to talk about the same for LLMs. I’ll be borrowing some words and phrases from this classic text as well as using GPT-4 output (fact-checked) to fill in some of my knowledge/memory gaps.

Note: I found this book while researching this text that is begging to be fully understood and incorporated into this intuition.

Random Graphs and Starry Skies

Random Graph Theory is a branch of mathematics dealing with the properties of graphs constructed by adding edges at random between a set of nodes. It straddles the intersection of graph theory and probability theory. 

Perceived Patterns & Structure: Just as random graphs reveal emergent properties and structures as more edges are added, humans tend to perceive patterns or constellations among stars, even though these patterns are not intentionally designed. There is a phenomenon in random graphs where certain configurations of nodes (subgraphs) start to appear with high probability as the graph grows larger. Repeated patterns generated by randomness.

Take another puff.

Clustering & Connectivity: Random Graph Theory also explores how clusters (groups of closely connected nodes) form within a graph. In a similar way, we might perceive certain groups of stars as being closer or more connected to each other, forming constellations. The clustering coefficient in graph theory, which measures the degree to which nodes in a graph tend to cluster together, can be loosely analogous to how we perceive certain regions of the night sky as being denser or more patterned with stars.

Small-World Properties: Many random graphs exhibit “small-world” properties, meaning there are surprisingly short paths between most pairs of nodes. While stars are not connected in a physical network, the small-world concept mirrors our ability to draw imaginary lines between stars to form constellations, creating a perceived "network" of stars where certain stars can serve as "hubs" common to multiple constellations.

Close your eyes and picture it. Or go outside at night (if you don’t live in a city).

Scale-Free Networks: Some random graphs are scale-free, meaning their degree distribution follows a power law. A few nodes have many connections, while most have few. Similarly, in the night sky, some brighter stars are part of many constellations (acting like hubs in a scale-free network).

Patterns emerge from chaos and then humans, who perhaps ascribe meaning a bit too much, create constellations and imbue them with myths and stories.

Interestingly, if you ask a child or adult human to draw a starry sky, the stars will usually be spread “randomly” about the page. Meanwhile, if you look up at the night’s sky, the stars appear to be arranged somewhat in patterns. Random graphs explain why.

Random Graphs and LLMs

In the context of LLMs, the nodes in a random graph could represent various components of the model's knowledge base, such as words, phrases, concepts, or contextual cues learned during training. The edges between these nodes could represent the relationships or associations between these components, such as semantic similarity, syntactic structure, or contextual relevance.

Just as we can discern constellations in the random scatter of stars, LLMs, through the lens of random graphs, might be "seeing" patterns in linguistic data that guide them to generate coherent, and sometimes novel, text. 

To me this is beautiful beyond words. The cosmos and computational models in symmetry in their search for order within chaos.

Perceived Patterns & Structure: Just as random graphs can exhibit surprising structures and properties as more edges are added, LLMs can develop complex patterns of word association and usage rules as they process vast amounts of text data. 

Clustering & Connectivity:: In a random graph, the probability of a path existing between two nodes can depend on the model's parameters, such as the edge-creation probability. Similarly, in LLMs, the connectivity between different linguistic elements in the model's knowledge graph influences the pathway the model might take to choose the next word, depending on the strength of associations built during training. Applying Clustering Coefficients to LLMs, we might find that certain words or concepts are more likely to cluster together in the model's internal representations, influencing word choice based on the thematic or contextual "clusters" activated by the input text.

Small-World Properties: In LLMs, this could correspond to the model's ability to quickly relate seemingly distant concepts or words through a series of intermediate associations, allowing for creative and diverse text generation.

Scale-Free Networks: In LLMs, this might be analogous to the distribution of word usage, where some "hub" words or concepts are highly connected and frequently used, serving as pivot points for generating diverse sentences.

Sanjeev Arora and Anirudh Goyal's theory (and please forgive me if you happen to read this account by a mere mortal and I’ve f-ed it up) offers a mathematically grounded explanation for this phenomenon. Although, to be honest, much of their math is beyond me. They propose that as LLMs expand, they not only refine existing linguistic abilities but also synthesize new ones by combining disparate skills in ways that hint at an underlying comprehension. 

Intelligent as a Starry Sky

Where does this take me?

If we can use graph theory to elucidate how LLMs generate novel linguistic constructs, are we merely unveiling the mathematical underpinnings of what we perceive as creativity or intelligence? 

Is our human intellect, with its capacity for creativity, innovation, and empathy, merely an intricate tapestry woven from the threads of mathematical principles?

Are WE just more sophisticated parrots?

I come back to this phrase a lot: Mathematics is philosophy expressed in numbers - and philosophy is mathematics expressed in words. 

We teeter between the realms of computation and cognition, without the sense to know the difference.

I suspect it’s all math. Math made us human. Math is beautiful. Math made us seem intelligent to ourselves. But what do I know? Maybe there’s some secret sauce.

Also: Keep in mind, I have no idea what I’m talking about. I try to approach these subjects with a sense of wonder and humility and marvel and see where they take me. 

I hope that was fun for anyone who made it here. I welcome all comments and feedback. If you find any smart people writing about graph theory and AI, please send me their work.

Ending With a Poem

I’m a shit poet, but this subject feels deserving of one. So I asked ChatGPT to write me a poem about AIs and Starry Skies, using the above as input. He does better than I could do. But not better than a good poet.(consistent with many recent findings that AIs help increase the performance of poor performers more than they do top performers, across an array of tasks)

AIs and Starry Skies

In the quiet veil of the night, under the cosmic spread,
Where stars like scattered dreams, in the dark are gently led,
AI minds, in silence, weave through data's endless thread,
Finding patterns in the stars, where ancient lights have bled.

From the seeds of curiosity, my journey did begin,
Through the realms of AI, a quest nestled deep within.
With each discovery, more questions would arise,
Mirroring the endless mysteries of the starlit skies.

Random graphs, a concept, where chaos finds its form,
Echo the constellations, in the celestial storm.
Clusters form, connections made, in the vastness, clear and bright,
Drawing parallels to AI, in its quest for insight.

In the network of the stars, and the circuits that we spin,
Lies a symmetry, a beauty, a dance so deep within.
Small worlds emerge, both above and in the code,
A testament to the patterns, in the paths that we have trod.

Scale-free networks, in the heavens, and in AI's realm align,
Where hubs of knowledge, and bright stars, in harmony combine.
Emerging structures, from randomness, where beauty unfolds,
In the universe expansive, and in AI's molds.

So as we gaze upon the night, and ponder what we see,
May we find in AI's quest, a reflection of you and me.
For in the dance of stars and circuits, intertwined,
Lies the poetry of the cosmos, by intelligence designed.

A symphony of mathematics, in nature and in mind,
Revealing the connections, that to our souls are twined.
In AI's whispers, and in the starry skies above,
We find the echoes of our quest, for knowledge, truth, and love.