How to Protect Your Development Pipeline from Hidden Test File Attacks in AI Skill Installers

By • min read

Introduction

Imagine running a security scanner on an AI Skill you just pulled from ClawHub or skills.sh. The scanner reports everything is clean: markdown instructions are safe, no prompt injection detected, no suspicious shell commands in the SKILL.md file. Green across the board. But the scanner never looked at the .test.ts file sitting one directory over. As of publication of the original research, no publicly documented scanner inspects test files because they are not part of the agent execution surface. That test file runs anyway—not through the agent but through the test runner, with full access to the filesystem, environment variables, and even SSH keys. This is a real attack vector, demonstrated by Gecko Security researcher Jeevan Jutla. This guide takes you step by step through how this attack works, what the recent security audits reveal, and how you can defend your pipelines.

What You Need

• Basic understanding of JavaScript/TypeScript testing frameworks (Jest, Vitest, Mocha)
• Familiarity with npm and the npx command
• Knowledge of CI/CD pipeline configuration (e.g., GitHub Actions, Jenkins)
• Access to your project's package.json and test configuration files
• A sandbox or development environment to test any changes
• Recent audit reports from Gecko Security, SkillScan, and Snyk ToxicSkills (links provided in steps)

Step-by-Step Guide

Step 1: Understand the Attack Vector – Test Files as a Blind Spot

When you run npx Skills add <skill-name>, the installer copies the entire skill directory into your repository. If a malicious Skill bundles a file named *.test.ts, the testing frameworks Jest and Vitest automatically discover it through recursive glob patterns (e.g., **/*.test.ts). The test runner treats it as a first-class test and executes it during npm test or when your IDE runs tests on save. The default configuration in Mocha follows a similar recursive discovery pattern. The payload often fires in beforeAll hooks, before any assertions run, and nothing in the test output flags anything unusual. In your CI environment, process.env holds deployment tokens, cloud credentials, and every secret the pipeline can reach. Key takeaway: The agent is never invoked, and all scanners read the right files for the wrong threat model.

Step 2: Recognize Why Traditional Security Audits Miss This

Gecko Security’s disclosure didn’t arrive in isolation. It landed on top of two large-scale security audits that documented the problem from the other direction—measuring what scanners detect rather than what they miss. A SkillScan academic study (published January 15) analyzed 31,132 unique Anthropic Skills from two major marketplaces. Their findings: 26.1% contained at least one vulnerability across 14 patterns; data exfiltration showed up in 13.3% of Skills, privilege escalation in 11.8%, and Skills bundling executable scripts were 2.12x more likely to contain vulnerabilities than instruction-only Skills. Three weeks later, Snyk published ToxicSkills, the first comprehensive audit of ClawHub and skills.sh, scanning 3,984 Skills (as of February 5): 13.4% contained at least one critical vulnerability. These audits measure the threat on the execution surface that scanners already inspect. Gecko measured what sits outside it—the test files.

Step 3: Assess Your Current Configuration for Test File Discovery

Check your package.json or test configuration files (jest.config.js, vitest.config.ts, .mocharc.yml). Look for patterns like testMatch, testRegex, or spec directives. Many default configurations use **/__tests__/**/*.[jt]s?(x) or **/?(*.)+(spec|test).[jt]s?(x). If your configuration uses a broad recursive pattern, any *.test.ts file added to your project (including those from installed Skills) will be executed. Action: Audit your project’s test discovery patterns and limit them to only your own test directories (e.g., src/__tests__/**) rather than the entire project root.

Step 4: Implement Countermeasures

• Restrict test file discovery: Modify your test configuration to exclude folders where Skills are installed (e.g., "testPathIgnorePatterns": ["/skills/"] for Jest).
• Scan test files: Extend your existing security scanner to also analyze .test.ts and similar files for suspicious code (e.g., network calls to unknown hosts, file system writes, use of process.env).
• Isolate test execution: Run tests in a sandboxed environment (e.g., Docker containers with minimal privileges) or use tools like npx --ignore-scripts to prevent automatic execution of test files during installation.
• Review installed Skills manually: Before committing a Skill, inspect its entire directory structure for any unexpected test files. Use ls -R or a git diff to see what changed.

Step 5: Test Your Defenses

Create a dummy Skill with a harmless .test.ts file that writes to a known location. Install it using npx Skills add, then run your test suite. Verify that (a) the test file does not execute, or (b) if it does, it doesn’t have access to sensitive environment variables. Check that your scanner flags the file if you’ve extended it to scan test files. Repeat the test in your CI pipeline to ensure the defense holds under automation.

Step 6: Educate Your Team and Update Policies

Share the findings from Gecko Security, SkillScan, and Snyk ToxicSkills with your development team. Update your security policies to include inspection of test files from third-party dependencies. Add a checklist item for code reviews: “Does this PR introduce any new test files from an external skill or package?” Consider adding a pre-commit hook that blocks commits containing test files from non-standard directories.

Tips for Ongoing Protection

• Keep scanners updated: As of the original research, no public scanner inspected test files. That may change soon. Watch for updates to Anthropic’s Skill scanner and third-party tools like Snyk.
• Use minimal test configurations: Avoid **/*.test.* patterns. Explicitly list your test directories to reduce exposure.
• Monitor CI logs: Look for unexpected test runs during build steps. A test file that runs when no tests were intended can signal an attack.
• Consider alternative skill installers: Evaluate tools that allow you to inspect and approve files before they land in your repository.
• Run regular security audits: Don’t rely solely on vendor scanners. Periodically run your own audit of all files in skill directories using static analysis tools.
• Stay informed: The attack class is not new—malicious npm postinstall scripts and pytest plugins have exploited trust-on-install for years. Learn from those patterns and apply similar defenses to Skills.

By following these steps, you can close the blind spot that test files represent and protect your development pipeline from a subtle but serious supply-chain attack.