Sentence to Object

This project coded in Python allows users to input a sentence or series of words. Those words are then put into an embedding model (all-MiniLM-L6-v2) and semantically parsed in multi-dimensional space, then reduced down to 3-dimensional space using PCA. This creates a point cloud of words in spatial relationship with one another. The pipeline then uses a series of different surface generation methods to convert these points into solid mesh models and generates an stl file that can be 3D printed. These objects are the physical representation of a Large Languages Model’s mathematical interpretation of the words. Each mesh vertex is a word (token) and each mesh edge is the vector length relationship between the words in 3D space. This project aims to reveal by making physically the black box process of how LLMs interpret language.  Slide show below is for this project and from a showcase presentation for a class entitled Agentic AI.

Surface generation methods include: 

Delaunay Triangulation: This method uses convex hull. It creates a shape that shrink wraps around the point cloud of words.  Convex hull only allows for convex forms so interior words are hidden inside the object.  Each vertex of the mesh object is a word, but not every word is represented by a vertex.  

Voronoi: The base of this shape in a Delaunay mesh. Each vertex or word from the Delaunay mesh acts as a seed. For every point in space it calculates which seed is closest and creates a polygon/polyhedron around that seed, which encompasses all space closer to it than any other seed.  These polygon/polyhedrons are the framework for the cell like structures and form half way between two seeds.  So the words or former vertices are located in the center of the cell structure openings.

swept tube: This method draws a line through space connecting all the input words in sentence order.  Each bend in the line is a word.

adaptive alpha: This method allows for convex as well as concave shapes.  It starts with the Delaunay method and then calcuates the space between word points. It creates convacity in the form when the word points are further apart (ie. when the words have less semantic similarity). Because a requirement is one single object, the alpha (α) variable is adaptive meaning it tests different sizes and picks the smallest size to fit between the word points before the object breaks apart into multiple meshes.  Becuase this process allows for concavity, more words to make it to the surface. As a result, these objects have more vertices. However every word is not alway represented as a vertex. It depends on the semantic similarity or vector length between the words.

3D printed objects below are from the same inputed words:

“Sprattling was one of the lucky audience members who snagged a coveted complimentary ticket with the help of her cousin, an artist who participated in the monthlong International Jazz Day festivities across the city. More than half of the roughly 3,000 seats were reserved for local creatives, students, educators and the general public, who registered online for a chance to receive admission.” (text taken from an email from WNYC)

This picture shows each mesh object type in the same spatial orientation. So you can compare the objects and see their similar underlying structure. On the adaptive alpha and delaunay shapes (two objects on the left) words are written on them so you can compare word locations.  “Complimentary”, “educator”, “online”, “recieved”, and “registered” are vertices on the Delaunay (second object). On the adaptive alpha object (first object) those words/ vertices are visible as well as the word “creative”. The concavity of the adaptive alpha releaves this word. It is also located closer to “educator” than “complinetary”, this is becuase they are loser in sematic meaning. 

3D printed objects. Mesh surface generation methods (right to left): adaptive alpha shape, Delauney Triangulation, Vornoi, sweep tube



Preview generated from pipeline. It shows the 3d mesh object with the words (tokens) mapped on to the objects. 
Video of adaptive alpha shape object




Slideshow of project explaining process: