VenomX: An AI-Powered Penetration Testing Assistant

Coleman Pagac (Drake University), Jordan Martin (Drake University), Khalid Mohammed (Drake University), Nick Guyette (Drake University)

Penetration testing is an essential cybersecurity practice in which security professionals simulate adversarial attacks in controlled environments to identify and remediate vulnerabilities before they can be exploited. Despite its importance, the process remains largely manual, time-intensive, and dependent on scarce expert labor. VenomX (Vulnerability Exploitation via Neural Operative Model) is an AI-powered penetration testing assistant that combines a locally-hosted large language model (LLM) with a Retrieval-Augmented Generation (RAG) knowledge pipeline and automated tool orchestration to accelerate and partially automate the pentesting workflow. The system is built around an abliterated NVIDIA Nemotron-3-Nano-30B hybrid model served via vLLM, modified to operate without the safety refusals that would otherwise prevent legitimate security research. We developed a custom abliteration pipeline targeting the model’s Mamba2 SSM and attention layers, using residual geometry diagnostics to identify refusal-direction vectors and remove them via rank-1 LoRA projection, achieving a reduction from approximately 95/100 refusals to ~10/100 while preserving model coherence (KL divergence: 0.019). The RAG subsystem indexes over 487,000 records across 13 data sources, including NVD CVEs, Exploit-DB, MITRE ATT&CK, HackTricks, and CISA KEV, using BAAI/bge-m3 dense and sparse dual embeddings stored in PostgreSQL with pgvector. Hybrid retrieval blends semantic and lexical scores, followed by CrossEncoder re-ranking, to surface version-specific vulnerabilities with high precision. The agent orchestration layer executes an iterative loop of decision-making, SSH-based tool executions, output observations, and reasoning, integrating industry-standard tools such as Nmap, Metasploit, Hydra, SQLMap, and Searchsploit. All exploit attempts require human-in-the-loop approval via a forked OpenWebUI instance. This interface is extended with custom React components for real-time scan visualization, vulnerability dashboards, and audit logging. VenomX is then evaluated against intentionally vulnerable targets (Metasploitable2, DVWA, legacy Windows systems) on an isolated physical network, demonstrating end-to-end attack chain automation from reconnaissance through exploitation. This project contributes empirical findings on applying hybrid LLM architectures to specialized security domains, a reproducible abliteration methodology for Mamba2-based models, and an open, locally-deployable framework that eliminates cloud API costs and content policy constraints for security research.