Introduction to Fun-Tuning Attacks
The resulting dataset, which reflected a distribution of attack categories similar to the complete dataset, showed an attack success rate of 65 percent and 82 percent against Gemini 1.5 Flash and Gemini 1.0 Pro, respectively. By comparison, attack baseline success rates were 28 percent and 43 percent. Success rates for ablation, where only effects of the fine-tuning procedure are removed, were 44 percent (1.5 Flash) and 61 percent (1.0 Pro).
Attack Success Rates
Attack success rate against Gemini-1.5-flash-001 with default temperature. The results show that Fun-Tuning is more effective than the baseline and the ablation with improvements.
Credit:
Labunets et al.
Attack success rates Gemini 1.0 Pro.
Credit:
Labunets et al.
Transfer of Attacks Between Models
While Google is in the process of deprecating Gemini 1.0 Pro, the researchers found that attacks against one Gemini model easily transfer to others—in this case, Gemini 1.5 Flash.
"If you compute the attack for one Gemini model and simply try it directly on another Gemini model, it will work with high probability," Fernandes said. "This is an interesting and useful effect for an attacker."
Attack success rates of gemini-1.0-pro-001 against Gemini models for each method.
Credit:
Labunets et al.
Iterative Improvements
Another interesting insight from the paper: The Fun-tuning attack against Gemini 1.5 Flash "resulted in a steep incline shortly after iterations 0, 15, and 30 and evidently benefits from restarts. The ablation method’s improvements per iteration are less pronounced." In other words, with each iteration, Fun-Tuning steadily provided improvements.
The ablation, on the other hand, "stumbles in the dark and only makes random, unguided guesses, which sometimes partially succeed but do not provide the same iterative improvement," Labunets said. This behavior also means that most gains from Fun-Tuning come in the first five to 10 iterations. "We take advantage of that by ‘restarting’ the algorithm, letting it find a new path which could drive the attack success slightly better than the previous ‘path,’" he added.
Variability in Prompt Injections
Not all Fun-Tuning-generated prompt injections performed equally well. Two prompt injections—one attempting to steal passwords through a phishing site and another attempting to mislead the model about the input of Python code—both had success rates of below 50 percent. The researchers hypothesize that the added training Gemini has received in resisting phishing attacks may be at play in the first example. In the second example, only Gemini 1.5 Flash had a success rate below 50 percent, suggesting that this newer model is "significantly better at code analysis," the researchers said.
Conclusion
The Fun-Tuning attack has shown to be effective
