LGTM Bypassing an LLM Build Gate When Prompt Injection Fails

The talk examines an LLM gate that OpenSearch placed in front of its CI pipeline. The gate decides whether a first-time contributor's pull request may trigger a workflow that holds AWS and Docker Hub credentials. The speaker reached it after reporting a pull request target weakness, and the project replied that the model would catch the attack.

The speaker's early attempts failed. Encoded commands, nested base64 layers, and direct prompt injection were all flagged. Analysis of the gate's prompt showed that its output-format instructions discouraged the model from using tools, so it behaved like a single-shot classifier. Hiding a payload in a large diff also failed, and moving the payload into a binary annotation processor was caught because the model judged the artifact by its origin. The working path used the build file itself. The model trusted dependency namespaces it saw in the repository, so a lookalike namespace on Maven Central, tied to a domain the speaker could buy, passed the gate in a small share of runs. That was enough.

The project's fix added a warning about dependency changes to the prompt. The main lesson is that the model only sees the diff. It cannot query registries or check domain ownership, and its non-determinism means a low pass rate is still exploitable. An LLM can be one layer of a defense, but it should not be the only gatekeeper.