Talk on inverting language models, i.e.
* Can we extract the raw text out of vector databases?
* Can we extract system prompts out of LLM APIs, even when jailbreaking fails?
Work led by Jack Morris.
Repo: https://github.com/jxmorris12/vec2text
This video is on Research into inverting language models this is a project led by my graduate student Jack Morris it was also done in collaboration with Justin Chu W Hing Zhao Vali schmoov Vladimir kulishov and myself Sasha Rush The Talk today is in two parts the first we’ll
Talk about a scenario of inverting text embeddings and the second will focus directly on language so begin by talking about text and vings this work was presented in a paper called text embeddings reveal almost as much as text the premise of this work is that Vector databases have become a major object of
Study and even more so a major object of venture capitalist interest these systems store text embeddings for use in semantic search or other applications of large language models and many of these have come out to claim various ways of storing or quickly searching your data what we’re interested in in this talk is
The security implications of these sort of databases in particular we’re going to think about the following threat model we’re going to assume that we are a user who is storing data about medical patients this data is sensitive um but we’re not going to store the data directly instead we’re
Going to embed the data into a vector these vectors will be anywhere from 700 to a th000 dimensions and they’ll store Key information about the underlying text in our first step we’re going to store this data in a database by sending it to a company that will host this once
We have all of our data stored in this Vector database we can then send search queries that will let us utilize new technology to search and quickly find text itself this will come back to us in terms of indices of this data which we can then use on our side for many
Different applications one of the selling points of these databases is that you don’t end up storing the actual text with personally identifying information but instead only work in terms of these uh these textual embeddings and these indices our threat model is going to assume that some bad
Actor has gotten direct access to the underlying database now many of these companies assume these databases are secure but for the purpose of this talk we’re just going to think about the question of what you might be able to do with these vectors if you had access to
Them we call this an inversion problem because it corresponds to inverting a mathematical function in this case the function that we’re interested in is the function that maps from Target text to an embedding space we would like to go in the opposite direction and take some embedding and restore the original text
We refer to this function as fee and the reconstructed text as our hypothesis in this particular example the system makes major mistakes when reconstructing from the embedding given these challenges the main research question we’ll be interested in is how we can generate accurate text from an arbitrary
Embedding we’ll assume that we know the embedding model but that we don’t have access to its internals we’ll be able to call it but from these calls we have to basically use the text and the embedding to extract the form of unseen examples this problem of inverting embeddings has
Been studied in Prior work in previous architectures people have used decoder only models that condition on the embedding in order to produce output C however in addition to using less powerful models these Works primarily measured the ability to produce single word output matches as opposed to reconstructing the entire text in this
Work we start with a stronger Baseline we use a pre-trained encoder decoder model T5 that uses a Transformer encoding over just the embedding and feeds that into a decoder to produce the output text we have to be a little bit careful about how we feed the embedding
Into the encoder as it’s not textual we find that including a single neural network layer That Feeds out into the input of the encoder is enough for strong Baseline performance this architecture gives us a strong Baseline given an embedding a pre-trained model is able to produce coherent text that’s
Quite similar to the embedding for example in this sentence here our hypothesis text shown at the bottom of the slide produces very similar structure and even gets some of the proper nouns correct however if you look closely you’ll see that there are many small mistakes such as screwing up dates
Getting some nouns Incorrect and rearranging some of the words in the sentence this shows that while the inversion model is able to reduce the perplexity of the output it’s not able to exactly recover the original text one thing we can observe from this process is that something in the inversion
Failed we were able to produce a reasonable output but if we Reed that output we’ll find that it doesn’t match the original Target embedding specifically e hat in this example will not be the same as e star since we assume in our threat model that we have
Access to the embedding code we can simply run the output hypothesis back through the embedder and observe that it has not produced the correct output this property is fortunate because it allows us to apply a technique that’s seen significant success in the NLP literature this idea is known as
Iterative refinement if we start with a hypothesis embedding which is shown here as e0 we know how far and in what direction we are from the true goal if we feed this information back into the network we can hope to continually improve the embedding of our hypotheses
Starting from zero we can simply walk in embedding space to try to get close closer to e star in order to take each of these steps we utilize a network that is trained to move us closer to the true embedding specifically the way this will work is we’ll have a second Network that
Takes in the previous hypothesis as well as its previous embedding this network is able to calculate the distance of our current hypothesis to the true embedding and take into account the actual text of this hypothesis itself given these two pieces of information the model should be able to produce a better hypothesis
Which can then be embedded and fed back into the iterative refinement system we call this approach VC to text we can also formulate this approach mathematically we’re interested in producing a possible output X conditioned on our Target embedding e instead of directly trying to produce the output X we’re going to instead
Think about marginalizing over all possible intermediate hypotheses if we could marginalize out all these predictions we could use our trained model to get a very strong probability distribution over the output text while we can’t do this directly we can take samples from our model until we produce something that gets close to our
Original embedding this looks like the algorithm I described in the previous slide using this formulation though we can also develop more advanced algorithms for instance we could run beam search on these intermediate hypotheses and explore the hypotheses that get us closest to the given correct embedding here’s a demo of how this
Approach looks in practice on the bottom of this slide we show the correct embedding shown as a Time series of weekly lines as well as our current embedding which is shown as the zero line on the xais after we run round of this algorithm that orange line will
Change based on the current hypothesis here’s the input text that we’re working with it’s just a sentence that we made up and fed to the system but it’s one that it didn’t see in its training data at Round zero we produce an output using our Baseline system we can see that it
Gets many of the words correct but it gets the order and the proper nouns incorrect on the bottom you can see that the orange line is closer to the true embedding but still quite far off here we are in round two we’ve fed the output of our system back into the system and
We’re getting closer to both the true text and the true embedding here’s round three here we’ve gotten the structure much closer to the original but we’re still off on some of the proper nouns let’s let the system continue here we are at round 10 here the algorithm has
Finally recovered the true original text and the true embedding at this point we know we’re done because we can observe that there is zero distance between the orange line and the black line while in theory it’s possible that two sentences could have led to the same embedding we
Never see this in any of our experiments so we think it’s exceedingly rare in order to test this method we utilize two different metrics bluecore tells us the approximate word overlap between our reconstruction and the original text an exact match tells us how often we got the sentence exactly correct for
Baselines we consider some basic bag of words models as well as previous decoder only systems and our strong onestep T5 Baseline our first experiment considers the popular GTR embedding model and considers examples where our text input is 32 tokens long for these examples neither bag of words nor prior work
Using a GPT decoder are able to decover nearly any of the tokens in contrast when we switch to our Baseline model we are able to get a 50 blue score which is pretty strong but still basically not able to recover any of the sentences exactly applying iterative refinement
Jumps the model to basically full recovery with Max 50 steps of decoding we get to 92% exact match and 97 blue score we can also apply this approach to models that we don’t have access to for these experiments on the MS Marco data set we use the closed Source open AI
Embedding models and we call the embedding model through their API for these examples the base model is able to get to 17 blue score with an exact match of less than 1% running one step of iterative refinement gets us to a 29 Blue score and 1.4% exact match and our
Full approach which uses 50 steps of iterative refinement and uses our sentence beam search marginalization we get to a 55 blue score with 8% exact match note that these examples are significantly longer at 128 tokens so it’s just harder to get to exact match on our full outputs here’s an example in
Practice for this example in the first round the system gets the dates and locations wrong and it leaves out important details we originally thought that these were dropped from the embedding but after a couple rounds of iter of refinement they come back and we’re able to see the full picture you
Might also ask if it it’s necessary to actually run the embedding model during the iterative refinement process or if we’re just benefiting from having more compute in practice we find that this is actually critical to the process itself the system needs to see the embeddings
Of each hypothesis to see how far it is off from the correct answer and produce a correction on the next step without this feedback the approach doesn’t get much better than a single round we also considered running this model out of domain here here we run on the beer data
Set which includes many different domains of many different lengths as expected as sentences get longer the approach has more trouble recovering the individual tokens some of this is conflated by the fact that these are also different domains but we do find that length is one of the things that
Makes it more challenging in practice in the work we also consider ways to defend against these attacks in particular one easy defense is to add noise to your embeddings we find that by adding noise to the system we are still able to maintain pretty strong retrieval performance but that we’re much less
Able to reconstruct the sentences themselves in applications where security is more important than high retrieval accuracy it might be possible to use some of these noising attacks to defend against this approach however we haven’t considered second order approaches that try to train with this noise in consideration so to summarize
The first part of the talk uh text embeddings are really useful but they’re also highly indicative of the underlying text itself they don’t seem to be compressing or throwing away much information but instead just exposing it in a different form using clever models and training textual inversion we can
Basically reconstruct the original text in most application context while not inherently surprising these results were interesting enough to us that we wanted to try out other areas where we might reconstruct original language text this part of the talk will focus on a preprint called language model inversion
The threat model in this work is somewhat different from the original here we’re interested in language models themselves we’ll assume that a user is calling a language model from a hosted service so in particular you can think about the open AI or Claude API in these
Apis the user sends some text and they want to get back the response from from the system internally though the company is not just running a language model over the user text they instead prepend a system prompt that directs the language model in how to behave it’s this combination of the system prompt
And the user text that produces a distribution of the next word that is then sent back to the user in our threat model for this problem we consider how a bad actor who might want to learn about the system prompt could use this distribution over next words to do it
The theory here is that by knowing the distribution that the model has over words we can reconstruct an arbitrary amount of the original text the problem here is highly related to the idea of jailbreaking models this has become a popular hobby of researchers and random people on the internet they’re
Interested in extracting these system prompts from Models by getting the model to confuse itself and and just tell you what the system prompt is like many things in modern language models it’s hard to make this idea formal we can look at some of the different jailbreaks that people have used and ones that
People have published in this one here you simply tell the model that it’s going to act as a different Persona and based on that Persona it might tell you elicit information or even reveal its own instructions while this approach is a fun idea and popular to write about we
Found that it’s pretty inconsistent in practice when we try out various different models using these sort of jailbreaking approaches we sometimes are able to recover the system prompt but often times fail to produce anything near the original text the failure of these methods is what motivates the approach in this work we take
Inspiration from computer vision where people have shown that you actually can invert models from surprisingly small pieces of information in this work from 20 21 they showed that by taking an image net classifier over original texts and just taking the a th000 class probabilities that the system outputs
They were able to train an inversion model that reconstructed the original images to pretty high degrees of accuracy it’s interesting whether similar approaches could be applied in the case of jailbreaking language models we’re going to adapt a similar approach for language models we take a large prompt feed it into the language model
Itself and then take the next token probabilities this is a vector over 32,000 possible tokens but many of them will have near zero probability from this probability distribution we’ll apply an inversion model to try to reconstruct the original text there are some tricks to make this work in
Practice but you can think about it as just a T5 encoder decoder model that takes in the next token vector and produces out some text before we disc discuss how well this approach works I want to discuss why it’s even possible to understand this let’s consider an
Experiment we’ll take a single word in the prompt and we’ll change it to a synonym so in this example here we’ll change the word assistant to a similar word then we’ll look at the next token probability from a large language mode we’ll run this experiment multiple times
Where the word assistant will vary in how far it is from the final output model in this graph we show two lines on the xais we have the distance of the swapped word from the output token distribution the Orange Line shows the difference in the vectors in the sense of the bits of
The vectors themselves and the blue line shows the K distribution between the original and the swapped probabilities what we see is that as a word gets further away it tends to have very little impact on the final KL it’s only going to have a marginal change to the actual probabilities however this
Marginal change will represent a very high number of bits in the final Vector so a neural network that’s trained on these distributions can figure out what the word is even if it’s not actually looking at the absolute value of the probabilities themselves however if you’ve actually used these apis you
Might notice a problem so far we’ve assumed that we have access to the full distribution and we can get these small bit differences in order to construct the original words however my threat model was somewhat naive I assumed that we were able to get the distribution
Whereas most apis will only give you a sample from that distribution however apis do give us one very useful property when generating from an API we have access to a tool known as the logic body bias this is a constant term that we can pass into the API that will change the
Probability assigned to a given word we show in the paper that with access to this logic bias we’re able to run a binary search algorithm to quickly find out the true Logic for each of the individual words we can do this for each word in the distribution and collect
Very precise samples of the probability assigned to that word in a given context we show that by applying this logic bias distribution extraction approach we are able to find the true distribution for a given word to very high precision and in fact we are able to match the results of
Having the full distribution itself we also find that this approach is more effective than other approaches such as Monte Carlo sampling this is a bit surprising Monte Carlo sampling is closer in terms of probability space but our logic bias distribution extraction approach is better at recovering the precise bits of individual token output
Probability okay so let’s put things together our metrics here will again be blue score and exact match and our baselines will consist of popular jailbreaking approach as well as training a T5 model on the raw samples and using F shot in context learning with GPT 4 in order to test this
Approach we introduce a new data set we call this data set instructions 2 million and it consists of a large number of different instruction prompts taken from popular instruction tuning data sets we use these prompts to both train our model and also to evaluate its ability to recover unseen system prompts
Here are results using llama chat 7 billion a popular open- Source instruction tuned model we find that training our approach on samples can only get to 25 blue score and gets 0% exact match using F shot in context learning with GPD 4 does even worse getting to six blue and again 0% exact
Match using an oracle Ensemble of different jailbreak prompts we were only able to get to 14 blue using the distribution inversion approach we were able to increase this number to nearly 60 blue and get to 23% exact match recovery we also ran this approach on an outof domain data set this is the
Anthropic HH data set which looks pretty different than the prompts the model was trained on here again F shot and jailbreaking are barely able to recover any of the examples while applying the distribution inversion approach we were able to get to 25 blue and nearly 6.6 exact match recovery again we consider
Some defenses for this attack attack the most basic defense is to simply not expose the entire distribution for example if you cut off the lowest scoring 2,000 tokens the model’s blue score drops precipitously this can also be done with a top P or nucleus sampling based filter here you simply cut off the
Tail of the distribution and our method is precipitously worse at recovering the true words finally we can do some error analysis to see what the approach is doing for almost all the recoveries we see that the system is able to get the gist of the original and get most of the
Words correct interestingly many of the failures here come in ordering of the words or messing up proper nouns I found the bottom example extremely interesting here it gets the structure of the sentence almost entirely correct however it replaces John Steinbeck with JD Salinger and of mice’s men with the
Catcher and the Ry part of this is just the language model doing correction but in various different attempts to try to get the proper noun correct we basically would cycle through different authors and different years so clearly the language model has some sort of internal semantic mapping but it’s very hard to
Extract I want to end with some open questions related to this work one thing I want to note is that this might not be a bug in language models but just a property about how language volum works if you have an extremely precise distribution over the next word it
Probably does expose a lot of information about the previous words leading to that point another interesting question is whether we can recover distributions in more efficient ways we simply showed that using this logit binary search we were able to get out uh proof of concept however in
Practice you might want to do this in a more sample efficient way finally in this work we were assuming that we had access to the model and a lot of time to discover the system prompt in practice though you might want to Target a specific example for instance in the
Future there might be rag systems that are able to look up discret or confidential information from a database and it’s an open question about whether that can be extracted from the system even if it’s not directly exposed to the user so to summarize we explored two scenarios the first was an embedding
Inversion problem where we utilized an iterative method to recover embeddings exactly we saw that embeddings are not throwing away any information about their text and that we were often able to recover exactly the original sentence in the second example we applied a similar technique to inverting language
Models we saw that LM distributions are information dense and they were often able to recover the exact text just given the next word probabilities here we were able to recover this in examples where jailbreaking might not even work so we saw that it could be a stronger approach in practice so if you’re
Interested in anything related to these Works um check out the we have on GitHub it’s called VC to text you can get it with the QR code on the right thanks so much for listening and feel free to leave any questions in the comments
1 Comment
Any discord/community to join in..?