lutz/MCP_CyberArk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit

Browse Source
This commit is contained in:
Lutz FInsterle
2026-03-29 19:51:51 +02:00
commit fd49e28d05
37 changed files with 6472 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

BIN
docs/HLD.docx Normal file
View File

Binary file not shown.

529
docs/HLD.md Normal file
View File

@@ -0,0 +1,529 @@
# High-Level Design
# MCP Privileged Access Service
**Version:** 1.1
**Date:** 2026-03-28
**Status:** Production-ready
---
## Table of Contents
0. [What is MCP? A Primer](#0-what-is-mcp-a-primer)
1. [Purpose & Scope](#1-purpose--scope)
2. [System Context](#2-system-context)
3. [Architecture Principles](#3-architecture-principles)
4. [Component Overview](#4-component-overview)
5. [Authentication & Authorization Model](#5-authentication--authorization-model)
6. [The Secret Handle Pattern](#6-the-secret-handle-pattern)
7. [Data Flow — Key Use Cases](#7-data-flow--key-use-cases)
8. [Deployment Architecture](#8-deployment-architecture)
9. [Technology Choices](#9-technology-choices)
10. [Security Architecture Summary](#10-security-architecture-summary)
11. [Future Roadmap](#11-future-roadmap)
---
## 0. What is MCP? A Primer
> **Learning note:** This section explains the MCP concept from first principles before we dive into this specific service. Skip to Section 1 if you are already familiar with the protocol.
### The core problem MCP solves
A large language model (LLM) like Claude is, at its heart, a text-in / text-out system. On its own it cannot *do* anything in the world — it can only describe what it would do. The challenge is: how do you give an AI assistant the ability to take real actions (read a file, query a database, run a command) in a controlled, auditable, and standardised way?
Before MCP, every team solved this differently. Some embedded shell calls directly in prompts; others built bespoke REST wrappers. There was no common contract between the AI and the tools it called.
**MCP (Model Context Protocol)** is Anthropic's open standard that defines exactly that contract.
---
### The three primitives
MCP defines three building blocks that a server can expose to a model:
| Primitive | What it is | Analogy |
|-----------|-----------|---------|
| **Tool** | A callable function the model can invoke | An API endpoint / RPC call |
| **Resource** | A piece of data the model can read | A file or database row |
| **Prompt** | A reusable prompt template | A macro or named query |
> **Learning note:** This service uses only **Tools**. Tools are the most important primitive for *agentic* use cases — cases where the model takes actions, not just answers questions. Resources and Prompts are useful but less common in automation pipelines.
---
### How a tool call works end-to-end
```
┌──────────────────────────────────────────────────────────────────────┐
│ USER "Check disk usage on web01" │
└────────────────────────────┬─────────────────────────────────────────┘
│ user message
▼
┌──────────────────────────────────────────────────────────────────────┐
│ CLAUDE (the model) │
│ Reads the list of available tools (JSON Schema descriptions). │
│ Decides: "I need ssh_execute to answer this." │
│ Emits a tool_use block in its response: │
│ { "name": "ssh_execute", │
│ "input": { "host": "web01", "command": "df -h", ... } } │
└────────────────────────────┬─────────────────────────────────────────┘
│ tool_use request (JSON-RPC over HTTP/SSE)
▼
┌──────────────────────────────────────────────────────────────────────┐
│ MCP SERVER (this service) │
│ Receives the JSON-RPC call. │
│ Executes the Python function ssh_execute(...). │
│ Returns a tool_result: { "content": "Filesystem Size Used…" } │
└────────────────────────────┬─────────────────────────────────────────┘
│ tool_result (text)
▼
┌──────────────────────────────────────────────────────────────────────┐
│ CLAUDE (the model) │
│ Incorporates the tool result into its context. │
│ Generates a final human-readable answer. │
└────────────────────────────┬─────────────────────────────────────────┘
│ assistant message
▼
┌──────────────────────────────────────────────────────────────────────┐
│ USER "web01: 18G used of 50G (36%)" │
└──────────────────────────────────────────────────────────────────────┘
```
> **Learning note:** The model **never executes code itself**. It only emits a structured request saying "please call this tool with these arguments." The MCP server is the only thing that touches real infrastructure. This separation is fundamental to safety — you can audit, rate-limit, and authorise every action at the server layer without modifying the model.
---
### Transport: SSE over HTTP
MCP uses **Server-Sent Events (SSE)** as its default transport. The client (Claude Code) opens a persistent HTTP connection to the server. The server streams JSON-RPC messages back as SSE events.
> **Learning note:** Why SSE and not WebSockets? SSE is unidirectional (server → client) and works over plain HTTP/1.1 with no protocol upgrade. This makes it firewall-friendly and easy to put behind standard reverse proxies like nginx. The request direction (client → server) still uses normal HTTP POST.
---
### FastMCP: the Python framework
Raw MCP requires implementing a JSON-RPC server, describing tools in JSON Schema, and handling SSE streams. **FastMCP** (Anthropic's Python library) removes all of that boilerplate:
```python
from mcp.server.fastmcp import FastMCP
mcp = FastMCP("my-server")
@mcp.tool(description="Add two numbers")
async def add(a: int, b: int) -> int:
return a + b
```
FastMCP introspects the Python type annotations and generates the JSON Schema automatically. The `@mcp.tool()` decorator registers the function — the function itself is just a normal `async def`.
> **Learning note:** This is exactly how all four MCP servers in this service are built. The tool functions (`get_credential`, `ssh_execute`, `ps_execute`, `db_query`) are plain Python async functions. The MCP protocol wrapping is invisible to the implementation code. You can call them directly in unit tests without any MCP machinery at all — which is why the test suite can be so simple.
---
### Context injection
FastMCP injects a `Context` object as the `ctx` parameter of every tool. You do not pass it yourself — the framework supplies it automatically when the tool is called over the MCP protocol.
```python
async def ssh_execute(host: str, command: str, ctx: Context, ...) -> str:
await ctx.info(f"Connecting to {host}") # progress notification to the caller
await ctx.error("Something went wrong") # error notification
```
`ctx.info()` and `ctx.error()` send notifications back to the client *during* tool execution, before the final result is returned. This is how Claude Code shows "Connecting to web01..." in its status bar while a long-running command is in progress.
> **Learning note:** In unit tests, `ctx` is a `MagicMock`. The tests assert on `ctx.info.call_args` and `ctx.error.call_args` to verify the right status messages were emitted — without any real MCP transport being involved.
---
### Multi-server composition
A single Python process can host **multiple independent MCP servers**, each mounted at a different URL path on a shared FastAPI application:
```
FastAPI app
├── /mcp/cyberark ← FastMCP("cyberark") — get_credential, list_safes
├── /mcp/ssh ← FastMCP("ssh") — ssh_execute
├── /mcp/powershell ← FastMCP("powershell")— ps_execute
└── /mcp/database ← FastMCP("database") — db_query
```
Claude Code is configured with four separate MCP server entries, each pointing to one of these paths. From Claude's perspective they appear as four separate servers, but they share a single process, a single secret store, and a single audit log stream.
> **Learning note:** Mounting multiple FastMCP instances on one FastAPI app via `app.mount(path, mcp.sse_app())` is the standard pattern for building multi-capability MCP services. The alternative — one process per server — would require inter-process communication to share the secret store, which adds complexity with no security benefit.
---
## 1. Purpose & Scope
The MCP Privileged Access Service enables Claude (Anthropic's AI assistant) to execute privileged operations on enterprise infrastructure — Linux servers via SSH, Windows servers via PowerShell/WinRM, and databases — using credentials managed by CyberArk Privileged Access Management (PAM).
**The fundamental security guarantee:**
> The AI model (Claude) **never sees the actual password** at any point in the workflow. Credentials are fetched from CyberArk, held in RAM behind an opaque token, and used directly for the target connection — all within the service boundary.
**Scope includes:**
- Retrieving credentials from CyberArk Central Credential Provider (CCP)
- Executing shell commands on Linux/Unix hosts via SSH
- Executing PowerShell scripts on Windows hosts via WinRM
- Running SQL queries on PostgreSQL, MySQL, and SQL Server databases
- Structured audit logging of all privileged operations
- API key authentication for Claude Code clients
**Scope excludes:**
- User interface or dashboard
- Credential rotation or lifecycle management (handled by CyberArk)
- Session recording (handled by CyberArk PSM if required)
- Multi-tenancy (single-tenant service per deployment)
---
## 2. System Context
```
┌──────────────────────────────────────────────────────────────────────┐
│ OPERATOR / SECURITY TEAM │
│ • Provisions CyberArk safes & AppID │
│ • Issues MCP API keys to Claude Code clients │
│ • Reviews structured audit logs │
└──────────────────────┬───────────────────────────────────────────────┘
│ configure
▼
┌──────────────────────────────────────────────────────────────────────┐
│ CLAUDE CODE (client) │
│ Claude Desktop / VS Code / CLI │
│ - Sends MCP tool calls over HTTPS with X-API-Key header │
│ - Receives tool results (output, exit codes, query rows) │
│ - NEVER receives actual passwords │
└──────────────────────┬───────────────────────────────────────────────┘
│ HTTPS + API Key (JSON-RPC / MCP protocol)
▼
┌──────────────────────────────────────────────────────────────────────┐
│ MCP PRIVILEGED ACCESS SERVICE (this system) │
│ ┌─────────────┐ ┌──────┐ ┌──────────┐ ┌────────┐ ┌─────────┐ │
│ │ CyberArk │ │ SSH │ │PowerShell│ │Database│ │ Auth + │ │
│ │ MCP │ │ MCP │ │ MCP │ │ MCP │ │ Audit │ │
│ └──────┬──────┘ └──┬───┘ └────┬─────┘ └───┬────┘ └─────────┘ │
│ │ │ │ │ │
│ └────────────┴────────────┴─────────────┘ │
│ Secret Store (RAM) │
└───┬──────────────┬──────────────┬──────────────┬─────────────────────┘
│ │ │ │
▼ ▼ ▼ ▼
┌───────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│CyberArk │Linux/Unix│ │ Windows │ │PostgreSQL│
│ CCP │ │ Hosts │ │ Hosts │ │ MySQL │
│(HTTPS)│ │ (SSH) │ │ (WinRM) │ │SQL Server│
└───────┘ └──────────┘ └──────────┘ └──────────┘
```
---
## 3. Architecture Principles
### P1 — Zero password exposure to the LLM
Passwords flow from CyberArk → RAM → target connection. At no stage does a password appear in an MCP tool response, log message, or error message. This is enforced in code, not by policy alone.
### P2 — Short-lived, single-use credential handles
A credential fetched from CyberArk is wrapped in a cryptographically random handle (`secret://` + 32-char hex). The handle:
- Expires after a configurable TTL (default 5 minutes)
- Is invalidated on first use (default `HANDLE_SINGLE_USE=true`)
- Lives only in process RAM — never written to disk or network
### P3 — Full audit trail
Every credential fetch, handle resolution, SSH execution, PowerShell execution, and database query is recorded in structured JSON logs. Passwords and output data are **never** included in audit events.
### P4 — Defence in depth
Multiple independent security layers:
1. Network: service only reachable from permitted IP ranges (firewall / VPC)
2. Transport: HTTPS with valid TLS certificate
3. Application: API key authentication on every request
4. Credential: CyberArk AppID + IP allowlist (or mTLS)
5. Handle: short TTL + single use
6. Code: `SecretStr` wrapper prevents accidental password serialisation
### P5 — Stateless compute, stateful secrets in RAM only
No database, no disk state. The secret store is an in-memory dict with an asyncio lock. Service restart invalidates all handles (safe failure mode — operators must re-fetch).
### P6 — Explicit over implicit
Every configuration value is explicit (`settings.*`), every dependency is injected at startup (lifespan), and every module imports only what it needs. No global mutable state except the two intentional singletons (`secret_store`, `cyberark_client`).
---
## 4. Component Overview
### 4.1 Foundation Layer
| Component | File | Role |
|-----------|------|------|
| Configuration | `config.py` | Single pydantic-settings model; reads from env / `.env` file |
| Secret Store | `secret_store.py` | In-RAM handle store with TTL, single-use, and background sweeper |
| Auth Middleware | `auth.py` | Starlette middleware; validates API key on all `/mcp/*` routes |
| Audit Logger | `audit.py` | Structured structlog events; one function per audit event type |
| Service Entry Point | `main.py` | FastAPI app assembly, lifespan wiring, MCP server mounting |
### 4.2 MCP Servers
| Server | Mount Path | Tool(s) | Protocol | Auth to target |
|--------|-----------|---------|----------|----------------|
| CyberArk | `/mcp/cyberark` | `get_credential`, `list_safes` | HTTPS REST | IP allowlist / mTLS |
| SSH | `/mcp/ssh` | `ssh_execute` | SSH (asyncssh) | Password from handle |
| PowerShell | `/mcp/powershell` | `ps_execute` | WinRM (pypsrp) | Password from handle |
| Database | `/mcp/database` | `db_query` | asyncpg / aiomysql / pyodbc | Password from handle |
Each MCP server is an independent `FastMCP` instance mounted as a sub-application on the shared FastAPI app. They share only two objects: `secret_store` (to resolve handles) and `settings` (configuration).
---
## 5. Authentication & Authorization Model
### Client → Service (inbound)
```
Claude Code client
│
│ HTTP request to /mcp/<server>/...
│ Header: X-API-Key: <key>
│ OR
│ Header: Authorization: Bearer <key>
│
▼
ApiKeyMiddleware
│
├── Path starts with /mcp/ ?
│ NO → pass through (health check, etc.)
│ YES → validate key against settings.mcp_api_keys
│ INVALID → 401 + audit log
│ VALID → continue to MCP handler
```
Multiple API keys are supported (comma-separated `MCP_API_KEYS`). Keys can be rotated by removing old keys and adding new ones, with no restart required if using a future key-reload mechanism.
### Service → CyberArk (outbound)
**Mode 1: IP Allowlist (current default)**
- The service makes HTTPS GET requests to the CCP REST API
- CyberArk trusts the caller based on source IP
- The AppID (`CYBERARK_APP_ID`) identifies the application in CyberArk policy
**Mode 2: mTLS (future)**
- A PFX certificate file is loaded at startup
- The TLS client certificate is attached to every CCP request
- CyberArk validates the certificate in addition to (or instead of) IP
### Service → Target Systems (outbound)
| Target | Auth method | Credentials from |
|--------|-------------|-----------------|
| SSH hosts | Password or key | Secret handle → `asyncssh.connect(password=...)` |
| WinRM hosts | NTLM / Basic | Secret handle → `WSMan(password=...)` |
| Databases | Native DB auth | Secret handle → driver connect call |
---
## 6. The Secret Handle Pattern
This is the central security innovation of the service. It solves the problem: *How does an AI model invoke privileged operations without ever knowing the password?*
```
Step 1 — Credential fetch
──────────────────────────
Claude calls: get_credential(safe="PROD-LINUX", object_name="svc_root")
Service:
1. Calls CyberArk CCP REST API
2. Receives { "UserName": "root", "Content": "P@ssword123", ... }
3. Calls secret_store.store("root", "P@ssword123")
→ stores in RAM as _Entry with a random 32-char hex handle_id
→ returns handle = "secret://a3f9c2e1b8d7..."
4. Returns to Claude: "Handle: secret://a3f9c2e1... TTL: 300s"
PASSWORD IS NEVER IN THIS RETURN VALUE
Step 2 — Privileged operation
──────────────────────────────
Claude calls: ssh_execute(host="server01", command="df -h",
secret_handle="secret://a3f9c2e1b8d7...")
Service:
1. Calls secret_store.resolve("secret://a3f9c2e1b8d7...")
→ checks TTL and single-use flag
→ if valid: returns ("root", "P@ssword123") and deletes handle
2. Calls asyncssh.connect("server01", username="root", password="P@ssword123")
3. Runs command, collects output
4. Deletes password variable (del password)
5. Returns: "Exit code: 0\nstdout:\n/dev/sda1 50G 10G 40G 20% /"
PASSWORD IS NEVER IN THIS RETURN VALUE
Step 3 — Handle is gone
────────────────────────
If Claude tries to reuse the same handle:
secret_store.resolve(...) raises KeyError("Handle already consumed")
→ Claude must call get_credential again for the next operation
```
**Handle lifecycle state machine:**
```
store()
CREATED ────────────────► ACTIVE
│
┌────────┴────────┐
│ │
resolve() TTL expired
(single_use=True) (sweeper task)
│ │
▼ ▼
CONSUMED EXPIRED
(deleted) (deleted)
```
---
## 7. Data Flow — Key Use Cases
### 7.1 SSH Command Execution
```
Claude CyberArk MCP SecretStore SSH MCP Linux Host
│ │ │ │ │
│ get_credential │ │ │ │
│────────────────►│ │ │ │
│ │ GET CCP REST │ │ │
│ │──────────────────────────────────────────────►│(CyberArk)
│ │◄─────────────────────────────────────────────-│
│ │ store(user,pw)│ │ │
│ │──────────────►│ │ │
│ │◄── handle ────│ │ │
│◄── handle ──────│ │ │ │
│ │ │ │ │
│ ssh_execute(handle) │ │ │
│─────────────────────────────────────────────────► │
│ │ │ resolve(handle│ │
│ │ │◄──────────────│ │
│ │ │──(user,pw)───►│ │
│ │ │ │ SSH connect │
│ │ │ │──────────────►│
│ │ │ │◄── output ───-│
│ │ │ │ del password │
│◄─────────────────────────────────────────────────output────────│
```
### 7.2 Database Query
Identical flow to SSH, substituting `db_query` for `ssh_execute` and the target database driver for asyncssh.
### 7.3 PowerShell Execution
The WinRM flow differs in one aspect: pypsrp is synchronous, so the call is offloaded to a thread-pool executor while the asyncio event loop continues serving other requests.
```
asyncio event loop Thread pool executor
────────────────── ────────────────────
resolve handle
await run_in_executor(None, _run_ps_sync, ...) ──────► _run_ps_sync()
[event loop free to handle other requests] WSMan()
RunspacePool()
ps.invoke()
◄────────────────────────────────────────────── return output
del password
return result
```
---
## 8. Deployment Architecture
### Recommended (Docker on a hardened VM)
```
┌──────────────────────────────────────────────┐
│ Hardened VM (e.g., Ubuntu 22.04) │
│ │
│ ┌─────────────────────────────────────┐ │
│ │ Docker container │ │
│ │ Image: mcp-privileged:1.0 │ │
│ │ User: mcpuser (non-root) │ │
│ │ Port: 8443 (internal) │ │
│ └──────────────┬──────────────────────┘ │
│ │ │
│ ┌──────────────▼──────────────────────┐ │
│ │ Reverse proxy (nginx / Caddy) │ │
│ │ TLS termination │ │
│ │ Port: 443 (external) │ │
│ └────────────────────────────────────── │
└──────────────────────────────────────────────┘
│
│ Firewall: only Claude Code source IPs allowed
```
### Network segmentation requirements
| Connection | Inbound to | Source | Port |
|-----------|------------|--------|------|
| Claude Code → Service | Service host | Claude Code client IPs | 443 (HTTPS) |
| Service → CyberArk CCP | CyberArk | Service host IP | 443 (HTTPS) |
| Service → SSH targets | Linux hosts | Service host IP | 22 (or custom) |
| Service → WinRM targets | Windows hosts | Service host IP | 5985/5986 |
| Service → Databases | DB servers | Service host IP | 5432/3306/1433 |
### Health check
`GET /health` returns `{"status": "ok"}` with no authentication. Suitable for load balancer and container health probes.
---
## 9. Technology Choices
| Technology | Choice | Rationale |
|-----------|--------|-----------|
| Web framework | FastAPI | Async-native, excellent OpenAPI support, Starlette middleware |
| MCP framework | FastMCP (mcp[server]) | Official Python MCP SDK; Streamable HTTP transport |
| HTTP client | httpx | Async, connection pooling, easy mock transport for tests |
| SSH | asyncssh | Pure-Python async SSH2; no subprocess dependency |
| WinRM | pypsrp | Python PowerShell Remoting Protocol; most complete WinRM library |
| PostgreSQL | asyncpg | Fastest async Postgres driver; native protocol |
| MySQL | aiomysql | Async MySQL driver |
| SQL Server | pyodbc | Standard ODBC; requires Microsoft ODBC Driver 18 on host |
| Config | pydantic-settings | Type-safe config; reads from env + `.env`; validates at startup |
| Logging | structlog | Structured JSON output; easy log shipping; context vars |
| Crypto | cryptography | PFX parsing for mTLS; well-maintained |
| Runtime | Python 3.11 | asyncio improvements, `tomllib`, `ExceptionGroup`, slots dataclasses |
| Container | Docker (multi-stage) | Small runtime image; non-root user; no build tools in production |
---
## 10. Security Architecture Summary
| Control | Implementation | Protects Against |
|---------|---------------|-----------------|
| TLS in transit | HTTPS everywhere (CCP, service) | Eavesdropping, MITM |
| API key auth | `ApiKeyMiddleware` on all `/mcp/*` | Unauthorised tool calls |
| CyberArk AppID | Registered in CyberArk policy | Unauthorised credential access |
| IP allowlist (CyberArk) | CyberArk trusted-net config | Rogue callers to CCP |
| mTLS (future) | PFX cert on CCP requests | Stronger caller identity |
| Secret handle | Opaque token, not password | Password exposure to LLM |
| Single-use handle | `handle_single_use=True` | Credential replay |
| TTL on handle | Default 300s | Handle leakage window |
| RAM-only storage | `SecretStore` dict, no disk I/O | Credential at-rest exposure |
| `SecretStr` wrapper | Pydantic `SecretStr` | Accidental log/repr of password |
| `del password` | Explicit deletion after use | Password in heap dumps |
| Audit log (no password) | structlog, explicit field list | Credential in log files |
| Non-root container | `USER mcpuser` in Dockerfile | Container escape impact |
| Output limits | 50 KB per stream, 1000 DB rows | Context flooding / DoS |
---
## 11. Future Roadmap
| Item | Priority | Description |
|------|----------|-------------|
| mTLS for CyberArk | High | Config already present; needs PFX cert provisioning |
| API key rotation without restart | Medium | Watch env file or use a config reload endpoint |
| SSH key-based auth | Medium | Support `asyncssh` with private key from CyberArk |
| Kerberos/NTLM for WinRM | Medium | Currently NTLM; Kerberos for domain environments |
| Connection pooling (SSH) | Low | Reuse SSH connections for repeated calls to same host |
| Multi-tenant API keys | Low | Map API keys to CyberArk AppIDs for key-per-team isolation |
| Metrics endpoint | Low | Prometheus `/metrics` for connection counts, handle stats |
| Session recording integration | Low | Forward SSH output to CyberArk PSM or a SIEM |

BIN
docs/LLD.docx Normal file
View File

Binary file not shown.

900
docs/LLD.md Normal file
View File

@@ -0,0 +1,900 @@
# Low-Level Design
# MCP Privileged Access Service
**Version:** 1.0
**Date:** 2026-03-28
**Status:** Production-ready
---
## Table of Contents
1. [Module Structure](#1-module-structure)
2. [Foundation Modules](#2-foundation-modules)
- 2.1 config.py
- 2.2 secret_store.py
- 2.3 auth.py
- 2.4 audit.py
- 2.5 main.py
3. [CyberArk MCP](#3-cyberark-mcp)
4. [SSH MCP](#4-ssh-mcp)
5. [PowerShell MCP](#5-powershell-mcp)
6. [Database MCP](#6-database-mcp)
7. [MCP Tool API Reference](#7-mcp-tool-api-reference)
8. [Data Models](#8-data-models)
9. [Configuration Reference](#9-configuration-reference)
10. [Error Handling Matrix](#10-error-handling-matrix)
11. [Audit Event Catalog](#11-audit-event-catalog)
12. [Test Strategy](#12-test-strategy)
---
## 1. Module Structure
```
src/mcp_privileged/
├── __init__.py
├── config.py ← All settings; read once at import time
├── secret_store.py ← In-RAM handle store + background sweeper
├── auth.py ← API key middleware
├── audit.py ← Structured log helpers
├── main.py ← FastAPI app assembly + lifespan
│
├── cyberark/
│ ├── __init__.py
│ ├── client.py ← CCP REST client (httpx)
│ └── server.py ← FastMCP server; get_credential, list_safes tools
│
├── ssh/
│ ├── __init__.py
│ └── server.py ← FastMCP server; ssh_execute tool
│
├── powershell/
│ ├── __init__.py
│ └── server.py ← FastMCP server; ps_execute tool
│
└── database/
├── __init__.py
└── server.py ← FastMCP server; db_query tool
```
**Dependency graph (no cycles):**
```
main.py
├── config.py (leaf)
├── audit.py ← config.py
├── auth.py ← config.py, audit.py
├── secret_store.py ← config.py, audit.py
├── cyberark/
│ ├── client.py ← config.py, audit.py
│ └── server.py ← cyberark/client.py, secret_store.py, audit.py
├── ssh/server.py ← config.py, secret_store.py, audit.py
├── powershell/server.py ← config.py, secret_store.py, audit.py
└── database/server.py ← config.py, secret_store.py, audit.py
```
---
## 2. Foundation Modules
### 2.1 config.py
**Class:** `Settings(BaseSettings)`
**Singleton:** `settings = Settings()` — imported everywhere as `from mcp_privileged.config import settings`
The settings object is created once when the module is first imported. If any required value is missing or invalid, pydantic raises a `ValidationError` at startup — fail fast.
#### Settings groups
**Service**
| Setting | Env var | Default | Type | Description |
|---------|---------|---------|------|-------------|
| `mcp_host` | `MCP_HOST` | `0.0.0.0` | `str` | Bind address for uvicorn |
| `mcp_port` | `MCP_PORT` | `8443` | `int` | Listen port |
| `mcp_api_keys_raw` | `MCP_API_KEYS` | *(required)* | `str` | Comma-separated API keys — access via the `mcp_api_keys` property |
**Secret Handle Store**
| Setting | Env var | Default | Type | Description |
|---------|---------|---------|------|-------------|
| `handle_ttl_seconds` | `HANDLE_TTL_SECONDS` | `300` | `int` (30–3600) | Handle expiry |
| `handle_single_use` | `HANDLE_SINGLE_USE` | `True` | `bool` | Invalidate on first resolve |
**CyberArk CCP**
| Setting | Env var | Default | Type | Description |
|---------|---------|---------|------|-------------|
| `cyberark_ccp_url` | `CYBERARK_CCP_URL` | — | `str` | Full CCP REST endpoint URL |
| `cyberark_app_id` | `CYBERARK_APP_ID` | — | `str` | AppID registered in CyberArk |
| `cyberark_verify_ssl` | `CYBERARK_VERIFY_SSL` | system CAs | `str` | `"false"`, `"true"`, or CA path |
| `cyberark_cert_pfx_path` | `CYBERARK_CERT_PFX_PATH` | `None` | `Path\|None` | mTLS client cert (PFX) |
| `cyberark_cert_pfx_password` | `CYBERARK_CERT_PFX_PASSWORD` | `None` | `str\|None` | PFX password |
**PowerShell / WinRM**
| Setting | Env var | Default | Type | Description |
|---------|---------|---------|------|-------------|
| `winrm_auth` | `WINRM_AUTH` | `ntlm` | `str` | `ntlm` or `basic` |
| `winrm_connect_timeout_seconds` | `WINRM_CONNECT_TIMEOUT_SECONDS` | `15` | `int` | WinRM connection timeout |
| `winrm_operation_timeout_seconds` | `WINRM_OPERATION_TIMEOUT_SECONDS` | `20` | `int` | WinRM operation timeout |
| `winrm_max_output_bytes` | `WINRM_MAX_OUTPUT_BYTES` | `51200` | `int` | Max bytes per output object |
**SSH**
| Setting | Env var | Default | Type | Description |
|---------|---------|---------|------|-------------|
| `ssh_known_hosts` | `SSH_KNOWN_HOSTS` | `~/.ssh/known_hosts` | `str` | Path or `"disable"` |
| `ssh_connect_timeout_seconds` | `SSH_CONNECT_TIMEOUT_SECONDS` | `10` | `int` | SSH connection timeout |
| `ssh_max_output_bytes` | `SSH_MAX_OUTPUT_BYTES` | `51200` | `int` | Max bytes per stdout/stderr |
**Database**
| Setting | Env var | Default | Type | Description |
|---------|---------|---------|------|-------------|
| `db_connect_timeout_seconds` | `DB_CONNECT_TIMEOUT_SECONDS` | `10` | `int` | DB connection timeout |
| `db_query_timeout_seconds` | `DB_QUERY_TIMEOUT_SECONDS` | `30` | `int` | Query execution timeout |
| `db_max_rows` | `DB_MAX_ROWS` | `1000` | `int` | Row result cap |
| `db_max_cell_bytes` | `DB_MAX_CELL_BYTES` | `1024` | `int` | Per-cell truncation threshold |
**Logging**
| Setting | Env var | Default | Type | Description |
|---------|---------|---------|------|-------------|
| `log_format` | `LOG_FORMAT` | `json` | `"json"\|"console"` | Output format |
| `log_level` | `LOG_LEVEL` | `INFO` | `"DEBUG"\|"INFO"\|..` | Minimum log level |
#### Validators
- `_parse_and_validate_api_keys` (model validator): validates that `MCP_API_KEYS` is non-empty and not equal to the default `"changeme"` — service refuses to start if either condition is violated. Raises `ValidationError` at import time (fail-fast).
- `mcp_api_keys` (property): splits `mcp_api_keys_raw` on commas, strips whitespace, returns `frozenset[str]`. Implemented as a `@property` (not a pydantic field) to avoid a name collision with the pydantic-settings env-var auto-mapping.
- `cyberark_ssl_verify`: maps `"false"` → `False`, `"true"` or `""` → `True`, anything else → path string
- `cyberark_cert_pfx_path`: empty string → `None`
- `_validate_pfx` (model validator): if PFX path is set, the file must exist and the password must be non-empty
---
### 2.2 secret_store.py
**Class:** `SecretStore`
**Singleton:** `secret_store = SecretStore()`
#### Internal data structure
```python
@dataclass(slots=True)
class _Entry:
handle_id: str # 32-char hex, the key in _store
username: str # plaintext (used as SSH/DB username)
password: SecretStr # pydantic SecretStr — prevents accidental str() exposure
created_at: float # time.monotonic() at creation
resolved: bool = False # set to True on first resolve
def is_expired(self, ttl: int) -> bool:
return (time.monotonic() - self.created_at) > ttl
```
```python
class SecretStore:
_store: dict[str, _Entry] # handle_id → entry
_lock: asyncio.Lock # all mutations are locked
```
#### Methods
**`async store(username, password) → str`**
1. Generate `handle_id = secrets.token_hex(16)` (32 hex chars, cryptographically random)
2. Create `_Entry(handle_id, username, SecretStr(password))`
3. Acquire lock, insert into `_store`
4. Return `"secret://" + handle_id`
**`async resolve(handle, resolved_by="unknown") → (str, str)`**
1. Parse handle → extract `handle_id` (raises `ValueError` if prefix wrong)
2. Acquire lock
3. Lookup entry — `KeyError` if not found
4. Check TTL — `KeyError("expired")` + delete if expired
5. Check `resolved` + `single_use` — `KeyError("already consumed")` if violated
6. Mark `entry.resolved = True`; delete if `single_use`
7. Release lock
8. Log `handle_resolved` audit event
9. Return `(entry.username, entry.password.get_secret_value())`
> `get_secret_value()` is the **only** intentional unwrap point in the entire codebase.
**`async revoke(handle) → bool`**
Explicit early revocation. Returns `True` if the handle existed.
**`async purge_expired() → int`**
Scans all entries and deletes expired ones. Called by the background sweeper every 60 seconds. Returns count of deleted entries.
#### Background sweeper
```python
async def _sweeper(store, interval_seconds=60):
while True:
await asyncio.sleep(interval_seconds)
count = await store.purge_expired()
```
Started in `main.py` lifespan as an `asyncio.Task`. Cancelled on shutdown.
---
### 2.3 auth.py
**Class:** `ApiKeyMiddleware(BaseHTTPMiddleware)`
```
Request arrives
│
▼
Does path start with "/mcp/"?
│
NO ├──────────────────────────────► pass through (health check etc.)
│
YES ▼
Extract key from headers:
1. X-API-Key: <value>
2. Authorization: Bearer <value>
│
▼
key in settings.mcp_api_keys ?
│
NO ├──────────────────────────────► 401 JSON + log_auth_failure()
│
YES ▼
call_next(request)
```
Key validation uses `hmac.compare_digest` in a non-short-circuiting loop over all configured keys, providing timing-safe comparison that prevents an attacker from inferring key length or prefix from response time differences:
```python
@staticmethod
def _is_valid_key(key: str) -> bool:
key_bytes = key.encode()
valid = False
for configured_key in settings.mcp_api_keys:
if hmac.compare_digest(key_bytes, configured_key.encode()):
valid = True # set flag, do NOT return early
return valid
```
The loop always iterates all keys (no `return True` inside the loop) so the response time does not leak how many keys are configured or how close a guess was.
---
### 2.4 audit.py
Wraps `structlog` with named functions so every audit event has a consistent schema. See [Section 11](#11-audit-event-catalog) for the full catalog.
**Configuration:** `configure_logging()` must be called once at startup (called in lifespan and in the `run()` entry point).
Processors pipeline:
```
merge_contextvars → add_logger_name → add_log_level → TimeStamper(iso)
→ StackInfoRenderer → ProcessorFormatter.wrap_for_formatter
→ [JSONRenderer | ConsoleRenderer]
```
Third-party loggers suppressed to WARNING: `uvicorn.access`, `asyncssh`, `pypsrp`.
---
### 2.5 main.py
**Function:** `create_app() → FastAPI`
Assembly sequence:
1. Create `FastAPI(lifespan=lifespan, docs_url=None, ...)` — docs disabled in production
2. Add `ApiKeyMiddleware`
3. Register `GET /health` route (no auth)
4. Import and mount four MCP servers:
- `cyberark_mcp` at `/mcp/cyberark`
- `ssh_mcp` at `/mcp/ssh`
- `powershell_mcp` at `/mcp/powershell`
- `database_mcp` at `/mcp/database`
**Lifespan (async context manager):**
```
startup:
configure_logging()
await cyberark_client.start() ← creates httpx.AsyncClient
sweeper_task = await start_sweeper(secret_store)
shutdown:
sweeper_task.cancel()
await sweeper_task ← wait for cancellation
await cyberark_client.stop() ← closes httpx.AsyncClient
```
**CLI entry point:** `mcp-privileged` → `mcp_privileged.main:run`
```python
def run():
configure_logging()
app = create_app()
uvicorn.run(app, host=settings.mcp_host, port=settings.mcp_port,
log_config=None, access_log=False)
```
---
## 3. CyberArk MCP
### 3.1 client.py — CyberArkCCPClient
**Singleton:** `cyberark_client = CyberArkCCPClient()`
#### Lifecycle
```python
await cyberark_client.start() # creates httpx.AsyncClient
await cyberark_client.stop() # closes httpx.AsyncClient
```
The `httpx.AsyncClient` is created once and reused for connection pooling. Timeouts: connect=5s, read=15s, write=5s, pool=5s.
#### `get_credential(app_id, safe, object_name) → Credential`
```
GET {CYBERARK_CCP_URL}?AppID={app_id}&Safe={safe}&Object={object_name}
```
Response parsing:
- HTTP 200 → `Credential` dataclass from JSON body
- HTTP 4xx/5xx → parse `ErrorCode`/`ErrorMsg` → raise `CyberArkError`
- Non-JSON body → raise `CyberArkError(status_code=...)`
- `httpx.ConnectError` → `CyberArkError("Cannot reach CCP")`
- `httpx.TimeoutException` → `CyberArkError("CCP request timed out")`
#### SSL modes
| Condition | `_build_ssl_context()` returns |
|-----------|-------------------------------|
| `cyberark_cert_pfx_path is None` | `settings.cyberark_ssl_verify` (bool or path) |
| PFX path set | `ssl.SSLContext` with client cert loaded |
For mTLS, the PFX is parsed with `cryptography`, cert+key are written to a `tempfile.mkstemp(suffix=".pem")` with `chmod 600`, loaded into the SSLContext, then the temp file is immediately deleted with `os.unlink()`.
#### Error codes
| Code | Meaning |
|------|---------|
| `APPAP004E` | AppID not found or not permitted |
| `APPAP006E` | Authentication failure (IP allowlist / AppID mismatch) |
| `APPAP007E` | Credential object not found in safe |
| `APPAP008E` | No password found for object |
| `APPAP009E` | Dual control pending approval |
| `APPAP010E` | Dual control approval timed out |
| `ITATS023E` | Object not found |
| `ITATS012E` | Safe not found |
### 3.2 server.py — CyberArk MCP
**Tools:** `get_credential`, `list_safes`
#### `get_credential(safe, object_name, ctx, app_id="")`
```
1. Resolve effective AppID (param or settings.cyberark_app_id)
2. ctx.info(...)
3. cyberark_client.get_credential(...)
→ on CyberArkError: ctx.error(...); raise
4. secret_store.store(credential.username, credential.password)
→ returns handle
5. log_credential_fetched(app_id, safe, object_name, handle_id, ttl, client_ip)
6. ctx.info("Credential retrieved. Handle issued...")
7. Return formatted string:
"Credential retrieved successfully.\n
Handle: secret://...\n
Username: ...\n
Address: ...\n
Platform: ...\n
TTL: 300 seconds\n
Use this handle with ssh_execute, ps_execute, or db_connect."
```
The return value is carefully crafted — it contains the handle (needed by Claude for the next step) plus metadata (username, address) to help Claude route the next call correctly, but **never** the password.
#### `list_safes(ctx, app_id="")`
Calls `cyberark_client.list_safes(app_id)`. Currently raises `NotImplementedError` (CCP has no native list-safes endpoint). The tool catches this and returns an informational message instead of raising.
---
## 4. SSH MCP
### 4.1 server.py
**Tool:** `ssh_execute`
#### Execution sequence
```
ssh_execute(host, command, secret_handle, ctx, port=22, username_override="", timeout_seconds=30)
│
├── secret_store.resolve(secret_handle, resolved_by="ssh")
│ KeyError → ctx.error(...); raise
│
├── if username_override: username = username_override
│
├── ctx.info("SSH connecting to ...")
│
├── _resolve_known_hosts(settings.ssh_known_hosts)
│ "disable" → None (no host key check, logs warning)
│ else → expanded path string
│
├── async with asyncssh.connect(host, port, username, password,
│ known_hosts, connect_timeout) as conn:
│ ├── result = await conn.run(command, timeout=timeout_seconds)
│ └── [exceptions caught — see error matrix]
│
├── del password (in finally block)
│
├── stdout = _truncate(result.stdout, ssh_max_output_bytes, "stdout")
├── stderr = _truncate(result.stderr, ssh_max_output_bytes, "stderr")
├── exit_code = result.exit_status ?? -1
│
├── log_ssh_executed(...)
├── ctx.info("SSH command completed ...")
│
└── return _format_result(host, command, exit_code, stdout, stderr)
```
#### Output format
```
Host: linux01.internal
Command: df -h
Exit code: 0
--- stdout ---
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 50G 10G 40G 20% /
--- stderr ---
(only present if non-empty)
```
#### Known hosts handling
| `ssh_known_hosts` value | Passed to asyncssh | Behaviour |
|------------------------|-------------------|-----------|
| `"disable"` | `None` | No host key verification (dev/lab only) |
| `"~/.ssh/known_hosts"` | `"/home/user/.ssh/known_hosts"` | Verify against file |
| `/etc/ssh/known_hosts` | `/etc/ssh/known_hosts` | Verify against file |
---
## 5. PowerShell MCP
### 5.1 server.py
**Tool:** `ps_execute`
#### Thread executor pattern
pypsrp is synchronous. The blocking WinRM call is wrapped in `asyncio.get_running_loop().run_in_executor(None, ...)`:
```python
loop = asyncio.get_running_loop()
output_lines, had_errors, error_records = await loop.run_in_executor(
None,
functools.partial(_run_ps_sync, host, port, username, password, script, use_ssl, timeout_seconds),
)
```
`None` uses the default `ThreadPoolExecutor`. The event loop remains responsive to other requests while WinRM is in progress.
#### `_run_ps_sync()` (thread worker)
```python
wsman = WSMan(
host, port=port, username=username, password=password,
ssl=use_ssl,
auth=settings.winrm_auth, # "ntlm" or "basic"
cert_validation=use_ssl, # only check cert when using HTTPS
connection_timeout=settings.winrm_connect_timeout_seconds,
operation_timeout=max(
timeout_seconds + 10,
settings.winrm_operation_timeout_seconds,
),
)
with RunspacePool(wsman) as pool:
ps = PowerShell(pool)
ps.add_script(script)
raw_output = ps.invoke()
had_errors = ps.had_errors
error_records = [str(e) for e in ps.streams.error]
```
Each item in `raw_output` is converted via `str()` and truncated to `winrm_max_output_bytes`.
#### Output format
```
Host: win01.internal
Script length: 43 chars
Had errors: False
--- output ---
WIN-SERVER-01
6.1.7601.65536
--- errors ---
(only present if had_errors or error_records is non-empty)
```
---
## 6. Database MCP
### 6.1 server.py
**Tool:** `db_query`
#### Driver dispatch
```python
async def _dispatch_query(db_type, host, port, database, username, password, query, timeout_seconds):
if db_type == "postgres": return await _query_postgres(...)
if db_type == "mysql": return await _query_mysql(...)
# mssql: run synchronous pyodbc in thread pool
return await loop.run_in_executor(None, partial(_query_mssql_sync, ...))
```
#### PostgreSQL (`asyncpg`)
```python
conn = await asyncpg.connect(host, port, user, password, database, timeout=connect_timeout)
rows = await conn.fetch(query, timeout=query_timeout)
columns = list(rows[0].keys())
data = [list(row.values()) for row in rows]
await conn.close()
```
#### MySQL (`aiomysql`)
```python
conn = await aiomysql.connect(host, port, user, password, db, connect_timeout)
async with conn.cursor() as cursor:
await asyncio.wait_for(cursor.execute(query), timeout=query_timeout)
columns = [col[0] for col in cursor.description]
rows = await cursor.fetchall()
conn.close()
```
#### SQL Server (`pyodbc` — sync)
```python
conn_str = (
"DRIVER={ODBC Driver 18 for SQL Server};"
f"SERVER={host},{port};DATABASE={database};UID={username};PWD={password};"
f"Connection Timeout={connect_timeout};"
)
with pyodbc.connect(conn_str, timeout=query_timeout) as conn:
cursor = conn.cursor()
cursor.execute(query)
columns = [col[0] for col in cursor.description]
rows = [list(row) for row in cursor.fetchall()]
```
If `pyodbc` is not importable (missing system ODBC driver), raises `RuntimeError` with installation instructions.
#### Row and cell limits
After query execution:
1. If `len(rows) > settings.db_max_rows`: truncate to `db_max_rows`, set `truncated=True`
2. For each cell in `_format_result`: `_cell_str()` truncates at `db_max_cell_bytes` UTF-8 bytes and appends `…`
#### Output format
```
Host: pg.internal
Database: prod (postgres)
Query length: 38 chars
Rows returned: 3
Elapsed: 12ms
id | name | email
----|---------|----------------
1 | Alice | alice@corp.com
2 | Bob | bob@corp.com
3 | Charlie | charlie@corp.com
```
If rows are capped: `Rows returned: 1000 (capped — more rows exist)`
---
## 7. MCP Tool API Reference
All tools follow the JSON-RPC 2.0 envelope defined by the MCP protocol. Parameters below are the tool-level parameters (inside `arguments`).
### `get_credential`
**MCP path:** `POST /mcp/cyberark/...`
| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `safe` | `string` | Yes | — | CyberArk Safe name |
| `object_name` | `string` | Yes | — | Credential object name in the Safe |
| `app_id` | `string` | No | `CYBERARK_APP_ID` | Override the service AppID |
**Returns:** Plain text with handle, username, address, platform, TTL.
**Errors:**
- `CyberArkError` — CCP returned an error (APPAP00xE etc.)
- `RuntimeError` — CyberArk client not started
---
### `list_safes`
**MCP path:** `POST /mcp/cyberark/...`
| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `app_id` | `string` | No | `CYBERARK_APP_ID` | AppID to list safes for |
**Returns:** Newline-separated list of Safe names, or informational message if not configured.
---
### `ssh_execute`
**MCP path:** `POST /mcp/ssh/...`
| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `host` | `string` | Yes | — | Hostname or IP |
| `command` | `string` | Yes | — | Shell command |
| `secret_handle` | `string` | Yes | — | Handle from `get_credential` |
| `port` | `integer` | No | `22` | SSH port |
| `username_override` | `string` | No | `""` | Override credential username |
| `timeout_seconds` | `integer` | No | `30` | Command timeout |
**Returns:** Formatted text with host, command, exit code, stdout, stderr.
**Errors:**
- `KeyError` — handle not found, expired, or already consumed
- `asyncssh.PermissionDenied` — authentication failure
- `asyncssh.DisconnectError` — SSH disconnection
- `asyncio.TimeoutError` — command timed out
- `OSError` — network error (connection refused, DNS failure)
---
### `ps_execute`
**MCP path:** `POST /mcp/powershell/...`
| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `host` | `string` | Yes | — | Hostname or IP |
| `script` | `string` | Yes | — | PowerShell script text |
| `secret_handle` | `string` | Yes | — | Handle from `get_credential` |
| `port` | `integer` | No | `5985` | WinRM port (5986 for HTTPS) |
| `use_ssl` | `boolean` | No | `false` | Use HTTPS for WinRM |
| `timeout_seconds` | `integer` | No | `60` | Script execution timeout |
| `username_override` | `string` | No | `""` | Override credential username |
**Returns:** Formatted text with host, script length, had_errors, output, error records.
**Errors:**
- `KeyError` — handle not found/expired/consumed
- Any exception from pypsrp (WinRM connection error, auth failure, etc.)
---
### `db_query`
**MCP path:** `POST /mcp/database/...`
| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `host` | `string` | Yes | — | Database server hostname |
| `database` | `string` | Yes | — | Database/schema name |
| `query` | `string` | Yes | — | SQL query text |
| `secret_handle` | `string` | Yes | — | Handle from `get_credential` |
| `db_type` | `string` | No | `"postgres"` | `"postgres"`, `"mysql"`, `"mssql"` |
| `port` | `integer` | No | `0` | 0 = use default for db_type |
| `username_override` | `string` | No | `""` | Override credential username |
| `timeout_seconds` | `integer` | No | `30` | Query timeout |
**Returns:** Text table with columns, rows, counts, elapsed time.
**Errors:**
- `ValueError` — unsupported `db_type`
- `KeyError` — handle not found/expired/consumed
- `asyncpg.PostgresError` — PostgreSQL error
- `aiomysql.Error` — MySQL error
- `pyodbc.Error` — SQL Server error
- `RuntimeError` — pyodbc not installed
---
## 8. Data Models
### Credential (CyberArk CCP response)
```python
@dataclass(frozen=True)
class Credential:
username: str # "svc_account"
password: str # raw password — stored in SecretStr immediately
address: str # "db.internal" — target host from CyberArk
safe: str # "PROD-DB"
folder: str # "Root"
object_name: str # "PROD-DB-svc_account"
platform_id: str # "Oracle", "UnixSSH", etc.
password_change_in_process: bool # True if CyberArk is rotating this credential
```
> `password_change_in_process=True` should trigger a warning — the credential may be mid-rotation.
### SecretStore entry
```python
@dataclass(slots=True)
class _Entry:
handle_id: str # 32 hex chars (key in _store dict)
username: str
password: SecretStr # pydantic SecretStr — str() returns "**********"
created_at: float # time.monotonic()
resolved: bool # True after first resolve
```
### Handle format
```
secret://a3f9c2e1b8d74f2c9e1a0b5d3c8f7e2a
└──────────────────────────────────┘
32-char lowercase hex = 128 bits of entropy
from secrets.token_hex(16)
```
---
## 9. Configuration Reference
Full `.env.example`:
```ini
# ── Service ───────────────────────────────────────────────────────────
MCP_HOST=0.0.0.0
MCP_PORT=8443
MCP_API_KEYS=key-for-claude-desktop,key-for-vscode
# ── Secret Handle Store ───────────────────────────────────────────────
HANDLE_TTL_SECONDS=300
HANDLE_SINGLE_USE=true
# ── CyberArk CCP ─────────────────────────────────────────────────────
CYBERARK_CCP_URL=https://cyberark.internal/AIMWebService/api/Accounts
CYBERARK_APP_ID=MCP-Privileged-Service
CYBERARK_VERIFY_SSL=/etc/ssl/certs/ca-certificates.crt
# ── CyberArk mTLS (leave empty for IP allowlist mode) ─────────────────
CYBERARK_CERT_PFX_PATH=
CYBERARK_CERT_PFX_PASSWORD=
# ── PowerShell / WinRM ────────────────────────────────────────────────
WINRM_AUTH=ntlm
WINRM_CONNECT_TIMEOUT_SECONDS=15
WINRM_OPERATION_TIMEOUT_SECONDS=20
WINRM_MAX_OUTPUT_BYTES=51200
# ── SSH ───────────────────────────────────────────────────────────────
SSH_KNOWN_HOSTS=~/.ssh/known_hosts
SSH_CONNECT_TIMEOUT_SECONDS=10
SSH_MAX_OUTPUT_BYTES=51200
# ── Database ──────────────────────────────────────────────────────────
DB_CONNECT_TIMEOUT_SECONDS=10
DB_QUERY_TIMEOUT_SECONDS=30
DB_MAX_ROWS=1000
DB_MAX_CELL_BYTES=1024
# ── Logging ───────────────────────────────────────────────────────────
LOG_FORMAT=json
LOG_LEVEL=INFO
```
---
## 10. Error Handling Matrix
| Layer | Exception | Handling | What Claude sees |
|-------|-----------|----------|-----------------|
| Auth middleware | Bad/missing key | 401 JSON response | `{"detail": "Invalid or missing API key"}` |
| SecretStore | `KeyError` (unknown) | Caught in tool, `ctx.error`, re-raised | MCP error response |
| SecretStore | `KeyError` (expired) | Same | MCP error response |
| SecretStore | `KeyError` (consumed) | Same | MCP error response |
| CyberArk CCP | `CyberArkError` | Caught in tool, `ctx.error`, re-raised | MCP error response with error code |
| SSH | `asyncssh.PermissionDenied` | `ctx.error`, re-raised | MCP error response |
| SSH | `asyncssh.DisconnectError` | `ctx.error`, re-raised | MCP error response |
| SSH | `asyncio.TimeoutError` | `ctx.error`, re-raised | MCP error response |
| SSH | `OSError` | `ctx.error`, re-raised | MCP error response |
| SSH | Non-zero exit code | NOT raised — returned in result | Normal result with `Exit code: N` |
| WinRM | Any exception from pypsrp | `ctx.error`, re-raised | MCP error response |
| WinRM | Script errors (`had_errors=True`) | NOT raised — returned in result | Normal result with `Had errors: True` |
| Database | `ValueError` (bad db_type) | Raised before credential access | MCP error response |
| Database | Driver exceptions | `ctx.error`, re-raised | MCP error response |
| Database | Row cap exceeded | NOT raised — result truncated | Normal result with `(capped)` note |
---
## 11. Audit Event Catalog
All events are emitted via structlog at `INFO` or `WARNING` level to the `audit` logger.
The logger is named `"audit"` — log shippers can filter on this name.
| Event | Level | When | Key fields |
|-------|-------|------|-----------|
| `credential_fetched` | INFO | CyberArk credential retrieved | `app_id, safe, object_name, handle_id, ttl_seconds, client_ip` |
| `handle_resolved` | INFO | Handle consumed by a tool | `handle_id, resolved_by, target_host, single_use_invalidated` |
| `handle_expired` | WARNING | Handle TTL exceeded or already consumed | `handle_id, reason` |
| `auth_failure` | WARNING | Invalid/missing API key | `client_ip, reason` |
| `cyberark_error` | ERROR | CCP returned error | `app_id, safe, object_name, status_code, error_code, message` |
| `ssh_executed` | INFO | SSH command completed | `handle_id, host, port, username, command, exit_code, elapsed_ms, client_ip` |
| `ps_executed` | INFO | PowerShell script completed | `handle_id, host, port, username, script_length, had_errors, elapsed_ms, client_ip` |
| `db_queried` | INFO | Database query completed | `handle_id, host, port, database, db_type, username, query_length, row_count, elapsed_ms, client_ip` |
**Fields intentionally absent from all events:**
- `password` (never)
- `secret_handle` (never — only `handle_id` which is non-reversible)
- stdout / stderr output (may contain sensitive data)
- SQL query text (logged only as `query_length`)
- PowerShell script text (logged only as `script_length`)
---
## 12. Test Strategy
### Test layout
```
tests/
├── conftest.py ← shared fixtures and mock helpers
├── test_auth.py ← API key middleware (FastAPI TestClient)
├── test_secret_store.py ← handle lifecycle (pure asyncio)
├── test_cyberark_client.py ← CCP HTTP client (httpx MockTransport)
├── test_ssh_server.py ← SSH tool (mock asyncssh.connect)
├── test_powershell_server.py ← PS tool (mock _run_ps_sync)
├── test_database_server.py ← DB tool (mock _dispatch_query)
└── test_integration.py ← end-to-end pipelines (all mocks combined)
```
### Test patterns
| Pattern | Used for | Why |
|---------|----------|-----|
| `httpx.MockTransport` | CyberArk client | Tests full HTTP response parsing without real CyberArk |
| `unittest.mock.patch` on transport layer | SSH, PowerShell, DB tools | Isolates MCP tool logic from network I/O |
| Real `secret_store` | All tool tests | Tests handle lifecycle end-to-end |
| `MagicMock` for `Context` | All tool tests | Tests `ctx.info` / `ctx.error` calls without MCP framework |
| `patch.object(settings, ...)` | Settings-sensitive tests | Overrides config for a test without process restart |
### Coverage targets
- Foundation modules: 100%
- CyberArk client: 100% (all HTTP response paths)
- MCP tools: ≥90% (happy path + all error paths)
- Integration flows: key pipelines (CyberArk→SSH, →PS, →DB)
- Known gap: real-system integration tests (require live CyberArk/WinRM/DB)
### Running tests
```bash
# All tests
python -m pytest tests/ -v
# With coverage
python -m pytest tests/ --cov=src/mcp_privileged --cov-report=term-missing
# Single module
python -m pytest tests/test_integration.py -v
```

BIN
docs/MANUAL.docx Normal file
View File

Binary file not shown.

966
docs/MANUAL.md Normal file
View File

@@ -0,0 +1,966 @@
# Operations Manual
# MCP Privileged Access Service
**Version:** 1.0
**Date:** 2026-03-28
**Audience:** System administrators, security engineers, DevOps teams
---
## Table of Contents
1. [Prerequisites](#1-prerequisites)
2. [CyberArk Prerequisites](#2-cyberark-prerequisites)
3. [Installation — Bare Metal / VM](#3-installation--bare-metal--vm)
4. [Installation — Docker](#4-installation--docker)
5. [Configuration Walkthrough](#5-configuration-walkthrough)
6. [SSH Host Key Setup](#6-ssh-host-key-setup)
7. [Windows WinRM Setup](#7-windows-winrm-setup)
8. [SQL Server ODBC Driver Setup](#8-sql-server-odbc-driver-setup)
9. [Claude Code Integration](#9-claude-code-integration)
10. [Usage Examples](#10-usage-examples)
11. [Monitoring & Log Events](#11-monitoring--log-events)
12. [Troubleshooting Guide](#13-troubleshooting-guide)
13. [Security Hardening Checklist](#14-security-hardening-checklist)
14. [Backup & Recovery](#15-backup--recovery)
15. [Upgrade Procedure](#16-upgrade-procedure)
---
## 1. Prerequisites
### System requirements
| Component | Minimum | Recommended |
|-----------|---------|-------------|
| OS | Ubuntu 22.04 / RHEL 9 | Ubuntu 22.04 LTS |
| CPU | 1 vCPU | 2 vCPU |
| RAM | 512 MB | 1 GB |
| Disk | 2 GB | 5 GB (for logs) |
| Python | 3.11 | 3.11 |
| Network | See firewall rules below | — |
### Network access required (outbound from service host)
| Destination | Port | Protocol | Purpose |
|-------------|------|----------|---------|
| CyberArk CCP | 443 | HTTPS | Credential retrieval |
| Linux target hosts | 22 | SSH | `ssh_execute` tool |
| Windows target hosts | 5985 or 5986 | HTTP/HTTPS | `ps_execute` tool (WinRM) |
| PostgreSQL servers | 5432 | TCP | `db_query` (postgres) |
| MySQL servers | 3306 | TCP | `db_query` (mysql) |
| SQL Server | 1433 | TCP | `db_query` (mssql) |
### Network access required (inbound to service host)
| Source | Port | Protocol | Purpose |
|--------|------|----------|---------|
| Claude Code clients | 443 | HTTPS | MCP tool calls |
| Load balancer / monitoring | 8443 | HTTP | Health check (if no TLS termination) |
---
## 2. CyberArk Prerequisites
Before deploying the service, complete the following in CyberArk.
### 2.1 Create an Application ID
1. In PVWA, navigate to **Applications** → **Add Application**
2. Set the application name to `MCP-Privileged-Service` (or your chosen value)
3. Under **Authentication**, add the service host's IP address to the **Allowed Machines** list
4. Save
### 2.2 Grant access to Safes
For each Safe containing credentials the service needs to retrieve:
1. Navigate to the Safe → **Members** → **Add Member**
2. Add `MCP-Privileged-Service` (the Application ID)
3. Grant permissions: **Retrieve accounts** (minimum)
4. Do **NOT** grant: Add, Update, Delete, Manage — principle of least privilege
### 2.3 Verify CCP is reachable
From the service host:
```bash
curl -k "https://cyberark.internal/AIMWebService/api/Accounts?AppID=MCP-Privileged-Service&Safe=TEST&Object=TEST-obj"
```
Expected responses:
- HTTP 200 — credential returned (safe and object exist)
- HTTP 404 `APPAP007E` — AppID valid but object not found (CCP is reachable and trusted)
- HTTP 403 `APPAP006E` — IP not in allowlist (add the service host IP to CyberArk)
- Connection refused — CCP URL is wrong or firewall is blocking
### 2.4 (Future) mTLS — Export client certificate
1. In CyberArk, generate or import a client certificate for the AppID
2. Export as PFX with a strong password
3. Copy the PFX file to the service host at a path like `/app/certs/mcp.pfx`
4. Set `chmod 400 /app/certs/mcp.pfx`
5. Set `CYBERARK_CERT_PFX_PATH=/app/certs/mcp.pfx` and `CYBERARK_CERT_PFX_PASSWORD=<password>` in `.env`
---
## 3. Installation — Bare Metal / VM
### 3.1 System packages
```bash
sudo apt-get update
sudo apt-get install -y python3.11 python3.11-venv python3.11-dev \
unixodbc unixodbc-dev ca-certificates
```
For SQL Server support (optional):
```bash
# Add Microsoft repository
curl https://packages.microsoft.com/keys/microsoft.asc | sudo apt-key add -
curl https://packages.microsoft.com/config/ubuntu/22.04/prod.list \
| sudo tee /etc/apt/sources.list.d/mssql-release.list
sudo apt-get update
sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18
```
### 3.2 Create service user
```bash
sudo useradd --system --no-create-home --shell /usr/sbin/nologin mcpuser
sudo mkdir -p /opt/mcp-privileged /opt/mcp-privileged/certs
sudo chown mcpuser:mcpuser /opt/mcp-privileged
```
### 3.3 Install the package
```bash
cd /opt/mcp-privileged
# Create and activate virtualenv
python3.11 -m venv .venv
source .venv/bin/activate
# Clone or copy source
# (assuming source is in /tmp/MCP_CyberArk)
pip install /tmp/MCP_CyberArk
# Verify
mcp-privileged --help
```
### 3.4 Configure
```bash
cp /tmp/MCP_CyberArk/.env.example /opt/mcp-privileged/.env
chmod 600 /opt/mcp-privileged/.env
nano /opt/mcp-privileged/.env
# Edit values — see Section 5
```
### 3.5 Configure SSH known_hosts
```bash
# Pre-populate known_hosts for all SSH target hosts:
sudo -u mcpuser ssh-keyscan linux01.internal linux02.internal >> \
/home/mcpuser/.ssh/known_hosts 2>/dev/null
# Or set SSH_KNOWN_HOSTS=/etc/ssh/known_hosts and populate there
```
### 3.6 Create systemd service
```bash
sudo tee /etc/systemd/system/mcp-privileged.service > /dev/null <<'EOF'
[Unit]
Description=MCP Privileged Access Service
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=mcpuser
Group=mcpuser
WorkingDirectory=/opt/mcp-privileged
EnvironmentFile=/opt/mcp-privileged/.env
ExecStart=/opt/mcp-privileged/.venv/bin/mcp-privileged
Restart=on-failure
RestartSec=5s
# Security hardening
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ReadWritePaths=/opt/mcp-privileged
CapabilityBoundingSet=
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable mcp-privileged
sudo systemctl start mcp-privileged
sudo systemctl status mcp-privileged
```
### 3.7 Reverse proxy (nginx)
```nginx
# /etc/nginx/sites-available/mcp-privileged
server {
listen 443 ssl;
server_name mcp.yourcompany.internal;
ssl_certificate /etc/ssl/certs/mcp.crt;
ssl_certificate_key /etc/ssl/private/mcp.key;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!aNULL:!MD5;
# Restrict to Claude Code client IPs (replace with real IPs)
allow 10.0.0.0/24;
deny all;
location / {
proxy_pass http://127.0.0.1:8443;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $host;
proxy_read_timeout 120s;
}
}
```
```bash
sudo ln -s /etc/nginx/sites-available/mcp-privileged /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx
```
---
## 4. Installation — Docker
### 4.1 Build the image
```bash
cd /path/to/MCP_CyberArk
docker build -t mcp-privileged:1.0 .
```
### 4.2 Create .env file
```bash
cp .env.example .env
chmod 600 .env
# Edit .env with your values
```
### 4.3 Run with Docker Compose
```bash
# Service only
docker compose up -d mcp-privileged
# Service + test databases (for integration testing)
docker compose --profile db up -d
```
### 4.4 Run with Docker (direct)
```bash
docker run -d \
--name mcp-privileged \
--restart unless-stopped \
-p 8443:8443 \
--env-file .env \
-v "$(pwd)/certs:/app/certs:ro" \
mcp-privileged:1.0
```
### 4.5 View logs
```bash
docker logs -f mcp-privileged
```
---
## 5. Configuration Walkthrough
Copy `.env.example` to `.env` and set each value:
### Mandatory values
```ini
# API keys — comma-separated, no spaces around commas
# Generate with: python3 -c "import secrets; print(secrets.token_hex(32))"
MCP_API_KEYS=abc123def456...,xyz789uvw012...
# CyberArk CCP URL — the full REST endpoint
CYBERARK_CCP_URL=https://cyberark.yourcompany.internal/AIMWebService/api/Accounts
# AppID registered in CyberArk (must match exactly — case-sensitive)
CYBERARK_APP_ID=MCP-Privileged-Service
```
### TLS verification
```ini
# Option 1: Use system CAs (default — works if CyberArk cert is signed by a trusted CA)
CYBERARK_VERIFY_SSL=true
# Option 2: Custom CA bundle (common for internal PKI)
CYBERARK_VERIFY_SSL=/etc/ssl/certs/internal-ca-bundle.crt
# Option 3: Disable (NEVER in production — dev/lab only)
CYBERARK_VERIFY_SSL=false
```
### Handle security
```ini
# How long a handle stays valid (seconds). Shorter = more secure.
# Operations that take < 30s: keep at 120-300s
# Long-running database imports: consider up to 600s
HANDLE_TTL_SECONDS=300
# Single-use enforces that each get_credential call is for one operation only.
# Set to false only if Claude needs the same credential for multiple parallel calls.
HANDLE_SINGLE_USE=true
```
### WinRM authentication
```ini
# ntlm — works for domain accounts, most common
# basic — works for local accounts but requires HTTPS (use_ssl=true in the tool call)
WINRM_AUTH=ntlm
```
### SSH known hosts
```ini
# Use the service user's known_hosts file (default)
SSH_KNOWN_HOSTS=~/.ssh/known_hosts
# Use a shared known_hosts for the whole service
SSH_KNOWN_HOSTS=/etc/mcp/ssh_known_hosts
# Disable host key checking (dev/lab ONLY — logs a warning on every connection)
SSH_KNOWN_HOSTS=disable
```
### Logging
```ini
# Production: use json for log shipping to SIEM
LOG_FORMAT=json
LOG_LEVEL=INFO
# Development: use console for human-readable output
LOG_FORMAT=console
LOG_LEVEL=DEBUG
```
---
## 6. SSH Host Key Setup
The service verifies SSH host keys against a `known_hosts` file. New hosts must be added before Claude can connect.
### Add a single host
```bash
# As the mcpuser (or root, then chown)
ssh-keyscan -H linux01.internal >> ~/.ssh/known_hosts
```
### Add multiple hosts from a list
```bash
cat hosts.txt | xargs ssh-keyscan -H >> ~/.ssh/known_hosts
```
Where `hosts.txt` contains one hostname per line.
### Using a shared known_hosts file
```bash
# Create shared file
sudo mkdir -p /etc/mcp
sudo ssh-keyscan -H linux01.internal linux02.internal db01.internal \
> /etc/mcp/ssh_known_hosts
sudo chown mcpuser:mcpuser /etc/mcp/ssh_known_hosts
sudo chmod 440 /etc/mcp/ssh_known_hosts
```
Then set `SSH_KNOWN_HOSTS=/etc/mcp/ssh_known_hosts` in `.env`.
### Verify a host key
```bash
ssh-keygen -F linux01.internal -f ~/.ssh/known_hosts
```
---
## 7. Windows WinRM Setup
### 7.1 Enable WinRM on Windows hosts
Run on each Windows target host (as Administrator):
```powershell
# Enable WinRM with default settings (HTTP, port 5985)
Enable-PSRemoting -Force
# Allow connections from the MCP service host IP
Set-Item WSMan:\localhost\Service\Auth\Basic -Value $true
winrm set winrm/config/client/auth '@{Basic="true"}'
# Allow specific IP in firewall (replace 10.0.0.5 with your service host IP)
New-NetFirewallRule -Name "WinRM-MCP" -DisplayName "WinRM for MCP Service" `
-Protocol TCP -LocalPort 5985 `
-RemoteAddress 10.0.0.5 -Action Allow
```
### 7.2 HTTPS WinRM (recommended for production)
```powershell
# On the Windows host — create HTTPS listener with a certificate
# (assumes cert is in the Local Machine store)
$cert = Get-ChildItem Cert:\LocalMachine\My | Where-Object { $_.Subject -like "*win01*" }
New-WSManInstance winrm/config/Listener `
-SelectorSet @{Transport="HTTPS"; Address="*"} `
-ValueSet @{CertificateThumbprint=$cert.Thumbprint}
# Open HTTPS WinRM port in firewall
New-NetFirewallRule -Name "WinRM-HTTPS-MCP" `
-Protocol TCP -LocalPort 5986 `
-RemoteAddress 10.0.0.5 -Action Allow
```
Then use `port=5986` and `use_ssl=true` in `ps_execute` tool calls.
### 7.3 Test WinRM from the service host
```bash
# Test HTTP WinRM connectivity (requires Python + pypsrp)
python3 -c "
from pypsrp.wsman import WSMan
from pypsrp.powershell import PowerShell, RunspacePool
wsman = WSMan('win01.internal', port=5985, username='domain\\\\svc_user',
password='P@ssword', ssl=False, auth='ntlm')
with RunspacePool(wsman) as pool:
ps = PowerShell(pool)
ps.add_script('hostname')
out = ps.invoke()
print(out)
"
```
---
## 8. SQL Server ODBC Driver Setup
Required for `db_query` with `db_type=mssql`.
### Ubuntu 22.04
```bash
curl https://packages.microsoft.com/keys/microsoft.asc | sudo apt-key add -
curl https://packages.microsoft.com/config/ubuntu/22.04/prod.list \
| sudo tee /etc/apt/sources.list.d/mssql-release.list
sudo apt-get update
sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18 unixodbc-dev
```
### Verify ODBC driver
```bash
odbcinst -q -d -n "ODBC Driver 18 for SQL Server"
# Should print the driver configuration
```
### Test SQL Server connectivity
```bash
python3 -c "
import pyodbc
conn = pyodbc.connect('DRIVER={ODBC Driver 18 for SQL Server};'
'SERVER=sql.internal,1433;DATABASE=master;'
'UID=sa;PWD=P@ssword;')
cur = conn.cursor()
cur.execute('SELECT @@VERSION')
print(cur.fetchone()[0])
"
```
---
## 9. Claude Code Integration
### 9.1 Configure MCP servers in Claude Code
Edit your Claude Code settings (usually `~/.claude/settings.json` or via `claude code config`):
```json
{
"mcpServers": {
"cyberark": {
"type": "http",
"url": "https://mcp.yourcompany.internal/mcp/cyberark",
"headers": {
"X-API-Key": "your-api-key-here"
}
},
"ssh": {
"type": "http",
"url": "https://mcp.yourcompany.internal/mcp/ssh",
"headers": {
"X-API-Key": "your-api-key-here"
}
},
"powershell": {
"type": "http",
"url": "https://mcp.yourcompany.internal/mcp/powershell",
"headers": {
"X-API-Key": "your-api-key-here"
}
},
"database": {
"type": "http",
"url": "https://mcp.yourcompany.internal/mcp/database",
"headers": {
"X-API-Key": "your-api-key-here"
}
}
}
}
```
### 9.2 Verify connectivity
In the Claude Code chat:
```
Check if the MCP servers are connected
```
Claude should report all four MCP servers (cyberark, ssh, powershell, database) as available tools.
### 9.3 Test with a simple operation
```
Using the PROD-LINUX safe, get the credential for svc_root on linux01.internal,
then run the command "whoami && uptime" on that host.
```
Claude should:
1. Call `get_credential(safe="PROD-LINUX", object_name="svc_root")`
2. Receive a handle
3. Call `ssh_execute(host="linux01.internal", command="whoami && uptime", secret_handle="secret://...")`
4. Return the output
---
## 10. Usage Examples
### Example 1: Check disk space on a Linux server
**User prompt to Claude:**
```
Get the root credential from the PROD-LINUX safe (object name: linux-root),
then check disk usage on server01.internal.
```
**What Claude does:**
1. `get_credential(safe="PROD-LINUX", object_name="linux-root")`
→ Returns: `Handle: secret://abc123... Username: root Address: server01.internal`
2. `ssh_execute(host="server01.internal", command="df -h", secret_handle="secret://abc123...")`
→ Returns:
```
Host: server01.internal
Command: df -h
Exit code: 0
--- stdout ---
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 50G 12G 38G 24% /
/dev/sdb1 200G 80G 120G 40% /data
```
---
### Example 2: Run a PowerShell script on Windows
**User prompt to Claude:**
```
Get the domain admin credential from WIN-SAFE (object: domain-admin),
then list all running services on win-server01.internal that are stopped.
```
**What Claude does:**
1. `get_credential(safe="WIN-SAFE", object_name="domain-admin")`
2. `ps_execute(host="win-server01.internal", script="Get-Service | Where-Object {$_.Status -eq 'Stopped'} | Select-Object Name, DisplayName", secret_handle="secret://...")`
→ Returns:
```
Host: win-server01.internal
Script length: 89 chars
Had errors: False
--- output ---
Name DisplayName
---- -----------
wuauserv Windows Update
XblGameSave Xbox Game Bar Saving Service
```
---
### Example 3: Query a database
**User prompt to Claude:**
```
Get the db_reader credential from DB-SAFE (object: pg-reader),
then count the orders placed in the last 24 hours in the prod PostgreSQL database
on pg.internal, database name: orders.
```
**What Claude does:**
1. `get_credential(safe="DB-SAFE", object_name="pg-reader")`
2. `db_query(host="pg.internal", database="orders", db_type="postgres", secret_handle="secret://...", query="SELECT COUNT(*) as orders_24h FROM orders WHERE created_at > NOW() - INTERVAL '24 hours'")`
→ Returns:
```
Host: pg.internal
Database: orders (postgres)
Query length: 84 chars
Rows returned: 1
Elapsed: 8ms
orders_24h
----------
1247
```
---
### Example 4: Multi-step workflow
**User prompt to Claude:**
```
I need to patch the Apache web servers in the PROD-LINUX safe.
For each of web01, web02, and web03:
1. Get the svc_admin credential
2. Run "sudo apt-get install --only-upgrade apache2 -y" on each host
3. Then check "apache2 -v" to confirm the version
```
**Note:** Because `HANDLE_SINGLE_USE=true`, Claude must call `get_credential` once per server (the handle is consumed by the first `ssh_execute`).
---
## 11. Monitoring & Log Events
### Log format (JSON)
```json
{
"event": "credential_fetched",
"logger": "audit",
"level": "info",
"timestamp": "2026-03-28T10:30:00.123Z",
"app_id": "MCP-Privileged-Service",
"safe": "PROD-LINUX",
"object_name": "linux-root",
"handle_id": "a3f9c2e1b8d74f2c",
"ttl_seconds": 300,
"client_ip": "10.0.0.50"
}
```
### Key events to alert on
| Event | Condition | Suggested alert |
|-------|-----------|-----------------|
| `auth_failure` | `reason=invalid_or_missing_api_key` | Any single occurrence |
| `auth_failure` | Rate > 5/minute from same IP | Possible brute-force |
| `cyberark_error` | `error_code=APPAP006E` | CyberArk allowlist may be wrong |
| `cyberark_error` | Rate > 10/hour | Possible misconfiguration |
| `handle_expired` | `reason=already_consumed` + high rate | Handle replay attempt |
| `ssh_executed` | `exit_code != 0` | Command failure — review |
| `ps_executed` | `had_errors=true` | Script error — review |
| Health check | No response within 10s | Service down |
### Log shipping
The service writes JSON logs to stdout. Use your standard log shipper:
**Filebeat:**
```yaml
- type: container
paths:
- /var/lib/docker/containers/*/*.log
processors:
- decode_json_fields:
fields: ["message"]
target: ""
```
**Splunk universal forwarder:**
Configure to tail the stdout log file or Docker container logs.
**Grafana Loki + promtail:**
```yaml
scrape_configs:
- job_name: mcp-privileged
docker_sd_configs:
- host: unix:///var/run/docker.sock
relabel_configs:
- source_labels: [__meta_docker_container_name]
regex: mcp-privileged
action: keep
```
---
## 12. Troubleshooting Guide
### Service fails to start
**Symptom:** `systemctl status mcp-privileged` shows `failed` or immediate exit.
**Check 1:** Configuration validation
```bash
cd /opt/mcp-privileged && source .venv/bin/activate
python3 -c "from mcp_privileged.config import settings; print('Config OK')"
```
If this fails, the error message shows which setting is invalid.
**Check 2:** PFX file (if mTLS is configured)
```bash
ls -la $CYBERARK_CERT_PFX_PATH
# Must exist and be readable by mcpuser
```
**Check 3:** Port in use
```bash
ss -tlnp | grep 8443
```
---
### 401 Unauthorized from Claude Code
**Cause:** API key mismatch between Claude Code settings and `MCP_API_KEYS`.
**Verify:**
```bash
# Check what keys are configured (value is obfuscated in logs)
grep MCP_API_KEYS /opt/mcp-privileged/.env
# Test with curl
curl -H "X-API-Key: your-key" https://mcp.yourcompany.internal/health
# Should return: {"status": "ok"}
```
---
### CyberArk error APPAP006E (authentication failure)
**Cause:** The service host's IP is not in the CyberArk allowlist for the AppID.
**Check:** What IP does CyberArk see?
```bash
# From the service host, check your outbound IP
curl https://api.ipify.org
# Or check your internal NAT gateway
```
**Fix:** In PVWA → Applications → `MCP-Privileged-Service` → Allowed Machines → Add the IP.
---
### CyberArk error APPAP007E (object not found)
**Cause:** The `safe` or `object_name` passed to `get_credential` does not exist in CyberArk.
**Check:**
- Spelling and case of Safe name (CyberArk is case-sensitive)
- Object name — this is the **Account name** (Name field), not the address or username
- The AppID has Retrieve permission on the Safe
---
### SSH connection fails: "Host key verification failed"
**Cause:** The target host's SSH fingerprint is not in the known_hosts file.
**Fix:**
```bash
ssh-keyscan -H linux01.internal >> ~/.ssh/known_hosts
# Or for the service user:
sudo -u mcpuser ssh-keyscan -H linux01.internal >> ~mcpuser/.ssh/known_hosts
```
**Quick diagnostic (dev only):** Temporarily set `SSH_KNOWN_HOSTS=disable` to confirm the issue is host key related, then fix properly.
---
### SSH connection fails: "Permission denied"
**Cause:** Wrong username/password, or password auth is disabled on the target host.
**Check:**
1. Verify the credential in CyberArk PVWA (test retrieval)
2. Confirm the target host allows password authentication: `PasswordAuthentication yes` in `/etc/ssh/sshd_config`
3. Confirm the account is not locked: `passwd -S <username>` on the target
---
### WinRM connection fails
**Symptom:** `ps_execute` returns a WinRM connection error.
**Check 1:** WinRM is running on the target
```powershell
# On the Windows host
Get-Service WinRM
winrm enumerate winrm/config/listener
```
**Check 2:** Firewall allows the connection
```powershell
# On the Windows host — test if port is open
Test-NetConnection -ComputerName localhost -Port 5985
```
**Check 3:** Auth method matches
- NTLM: works for domain accounts and most setups
- Basic: requires `WINRM_AUTH=basic` in `.env` AND `use_ssl=true` in the tool call (Basic auth over HTTP is rejected by WinRM by default)
---
### Database connection fails
**PostgreSQL:**
```bash
# Test from service host
psql -h pg.internal -U db_user -d mydb -c "SELECT 1"
```
**MySQL:**
```bash
mysql -h mysql.internal -u db_user -p -e "SELECT 1"
```
**SQL Server (ODBC):**
```bash
isql -v "DRIVER={ODBC Driver 18 for SQL Server};SERVER=sql.internal,1433;DATABASE=master" \
sa "P@ssword"
```
If `pyodbc` fails with `ImportError: libodbc.so.2: cannot open shared object file`:
```bash
sudo apt-get install -y unixodbc
```
---
### Handle expired / already consumed
**Symptom:** Tool returns `KeyError: Handle expired` or `Handle already consumed`.
**Causes:**
- The TTL elapsed between `get_credential` and the tool call → increase `HANDLE_TTL_SECONDS`
- `HANDLE_SINGLE_USE=true` and Claude tried to reuse the handle → normal behaviour; Claude should call `get_credential` again
- Clock skew on the service host (TTL uses `time.monotonic()`, so clock skew does not affect it)
---
## 13. Security Hardening Checklist
Use this checklist before production deployment.
### Network
- [ ] Service host is in a restricted network segment (not accessible from general office network)
- [ ] Firewall rules allow only approved Claude Code client IPs to reach port 443
- [ ] Service host can only reach: CyberArk CCP, target SSH hosts, WinRM hosts, DB servers — no internet
- [ ] Reverse proxy handles TLS termination with a valid internal CA certificate
### Service configuration
- [ ] `MCP_API_KEYS` is set to strong random keys (minimum 32 chars each)
- [ ] Default key `changeme` is NOT present in `MCP_API_KEYS`
- [ ] `HANDLE_SINGLE_USE=true` (default)
- [ ] `HANDLE_TTL_SECONDS` ≤ 300 (5 minutes)
- [ ] `CYBERARK_VERIFY_SSL` is **not** set to `false`
- [ ] `SSH_KNOWN_HOSTS` is **not** set to `disable`
- [ ] `LOG_FORMAT=json` (for log shipping)
- [ ] `.env` file has `chmod 600` and is owned by the service user
### CyberArk
- [ ] AppID has only Retrieve permission on Safes (no Add/Update/Delete)
- [ ] IP allowlist is restricted to the service host IP only
- [ ] A dedicated AppID is used for this service (not shared with other applications)
### Docker
- [ ] Container runs as non-root (`USER mcpuser` in Dockerfile — already done)
- [ ] Secrets are passed via `--env-file`, not `-e PASSWORD=...` in docker run
- [ ] Docker socket is not mounted into the container
- [ ] Image is built from official Python base image (verified digest)
### Operating system
- [ ] OS is patched and on a supported LTS release
- [ ] Service runs as a dedicated non-root user (`mcpuser`)
- [ ] systemd unit has `NoNewPrivileges=yes` and `ProtectSystem=strict`
- [ ] Log rotation is configured for stdout logs
- [ ] auditd or similar is monitoring privileged operations
---
## 14. Backup & Recovery
The service is **stateless**: no persistent data is stored on disk.
- **Configuration:** The only file that needs backing up is `.env`. Store it in your secrets management system (HashiCorp Vault, AWS Secrets Manager, etc.), not in a generic file backup.
- **Certificates:** Back up PFX files and known_hosts files to your PKI or secrets vault.
- **Recovery:** To restore after a host failure, provision a new VM, install the package, and restore `.env` + certificates. All handles in RAM are lost (no active handles = fail-safe state; users must call `get_credential` again).
- **RTO:** < 5 minutes (container restart or new VM + `.env` restore).
- **RPO:** 0 (no data to lose — the service holds no persistent state).
---
## 15. Upgrade Procedure
### Minor upgrade (no config changes)
```bash
# Docker
docker pull mcp-privileged:1.1
docker compose up -d mcp-privileged
# Bare metal
cd /opt/mcp-privileged && source .venv/bin/activate
pip install --upgrade /path/to/new/mcp_privileged-1.1.tar.gz
sudo systemctl restart mcp-privileged
```
Active handles are lost on restart (they expire within TTL anyway). Notify users if the restart window > 5 minutes.
### Major upgrade (config changes)
1. Read the release notes — check for new required env vars
2. Test in a staging environment first
3. Update `.env` with new required values
4. Follow the minor upgrade steps above
5. Monitor logs for errors in the first 10 minutes
### Rollback
```bash
# Docker — roll back to previous image tag
docker compose down
docker run --name mcp-privileged mcp-privileged:1.0 ...
# Bare metal
pip install mcp_privileged==1.0
sudo systemctl restart mcp-privileged
```

343
docs/md_to_docx.py Normal file
View File

@@ -0,0 +1,343 @@
"""
Convert the three MCP documentation Markdown files to Word (.docx) format.
Handles:
- Heading levels 1–4
- Bold (**text**) and inline code (`text`)
- Fenced code blocks (``` ... ```)
- Tables (| col | col |)
- Unordered lists (- item, * item)
- Ordered lists (1. item)
- Horizontal rules (---)
- Blank lines → paragraph spacing
Run:
python docs/md_to_docx.py
Produces:
docs/HLD.docx
docs/LLD.docx
docs/MANUAL.docx
"""
from __future__ import annotations
import re
from pathlib import Path
from docx import Document
from docx.enum.text import WD_ALIGN_PARAGRAPH
from docx.oxml.ns import qn
from docx.oxml import OxmlElement
from docx.shared import Inches, Pt, RGBColor
# ── Colour palette ────────────────────────────────────────────────────────────
DARK_BLUE = RGBColor(0x1F, 0x49, 0x7D) # heading 1
MID_BLUE = RGBColor(0x2E, 0x74, 0xB5) # heading 2
STEEL_BLUE = RGBColor(0x1F, 0x78, 0xB4) # heading 3
DARK_GREY = RGBColor(0x40, 0x40, 0x40) # body text
CODE_BG = RGBColor(0xF2, 0xF2, 0xF2) # code block shading
TABLE_HEAD = RGBColor(0x1F, 0x49, 0x7D) # table header background
TABLE_EVEN = RGBColor(0xEA, 0xF2, 0xFF) # alternating row colour
# ── Helpers ───────────────────────────────────────────────────────────────────
def _shade_cell(cell, colour: RGBColor) -> None:
"""Apply a solid background fill to a table cell."""
tc = cell._tc
tcPr = tc.get_or_add_tcPr()
shd = OxmlElement("w:shd")
shd.set(qn("w:val"), "clear")
shd.set(qn("w:color"), "auto")
shd.set(qn("w:fill"), f"{colour[0]:02X}{colour[1]:02X}{colour[2]:02X}")
tcPr.append(shd)
def _set_cell_border(cell, **kwargs) -> None:
"""Set borders on a table cell."""
tc = cell._tc
tcPr = tc.get_or_add_tcPr()
tcBorders = OxmlElement("w:tcBorders")
for side in ("top", "left", "bottom", "right", "insideH", "insideV"):
if side in kwargs:
border = OxmlElement(f"w:{side}")
for attr, val in kwargs[side].items():
border.set(qn(f"w:{attr}"), val)
tcBorders.append(border)
tcPr.append(tcBorders)
def _apply_inline(run, text: str) -> None:
"""Set run text, detecting and stripping bold/inline-code markers."""
run.text = text
def _parse_inline(para, text: str) -> None:
"""
Parse a line of text for inline Markdown:
**bold** → bold run
`code` → monospace run
plain → normal run
Adds runs to the given paragraph.
"""
pattern = re.compile(r'(\*\*[^*]+\*\*|`[^`]+`)')
parts = pattern.split(text)
for part in parts:
if not part:
continue
if part.startswith("**") and part.endswith("**"):
run = para.add_run(part[2:-2])
run.bold = True
elif part.startswith("`") and part.endswith("`"):
run = para.add_run(part[1:-1])
run.font.name = "Courier New"
run.font.size = Pt(9)
run.font.color.rgb = RGBColor(0xC0, 0x39, 0x2B)
else:
run = para.add_run(part)
def _add_heading(doc: Document, text: str, level: int) -> None:
"""Add a styled heading, stripping any leading '#' symbols."""
clean = re.sub(r"^#+\s*", "", text).strip()
# Remove anchor links like {#section-name}
clean = re.sub(r"\s*\{#[^}]+\}", "", clean)
para = doc.add_heading(clean, level=level)
run = para.runs[0] if para.runs else para.add_run(clean)
if level == 1:
run.font.color.rgb = DARK_BLUE
run.font.size = Pt(20)
elif level == 2:
run.font.color.rgb = MID_BLUE
run.font.size = Pt(15)
elif level == 3:
run.font.color.rgb = STEEL_BLUE
run.font.size = Pt(12)
else:
run.font.color.rgb = DARK_GREY
run.font.size = Pt(11)
run.bold = True
def _add_code_block(doc: Document, lines: list[str]) -> None:
"""Add a shaded monospace code block."""
para = doc.add_paragraph()
para.paragraph_format.left_indent = Inches(0.3)
para.paragraph_format.space_before = Pt(4)
para.paragraph_format.space_after = Pt(4)
# Add shading via XML
pPr = para._p.get_or_add_pPr()
shd = OxmlElement("w:shd")
shd.set(qn("w:val"), "clear")
shd.set(qn("w:color"), "auto")
shd.set(qn("w:fill"), "F2F2F2")
pPr.append(shd)
text = "\n".join(lines)
run = para.add_run(text)
run.font.name = "Courier New"
run.font.size = Pt(8.5)
run.font.color.rgb = RGBColor(0x1A, 0x1A, 0x1A)
def _add_table(doc: Document, rows: list[list[str]]) -> None:
"""Add a formatted table. First row is treated as the header."""
if not rows:
return
col_count = max(len(r) for r in rows)
# Normalise row lengths
rows = [r + [""] * (col_count - len(r)) for r in rows]
table = doc.add_table(rows=len(rows), cols=col_count)
table.style = "Table Grid"
for row_idx, row_data in enumerate(rows):
row = table.rows[row_idx]
for col_idx, cell_text in enumerate(row_data):
cell = row.cells[col_idx]
clean = cell_text.strip().strip("`")
para = cell.paragraphs[0]
para.paragraph_format.space_before = Pt(2)
para.paragraph_format.space_after = Pt(2)
if row_idx == 0:
# Header row
_shade_cell(cell, TABLE_HEAD)
run = para.add_run(clean)
run.bold = True
run.font.color.rgb = RGBColor(0xFF, 0xFF, 0xFF)
run.font.size = Pt(9)
else:
if row_idx % 2 == 0:
_shade_cell(cell, TABLE_EVEN)
_parse_inline(para, clean)
for run in para.runs:
run.font.size = Pt(9)
doc.add_paragraph() # spacing after table
def _add_list_item(doc: Document, text: str, level: int, ordered: bool,
counter: int) -> None:
"""Add a bullet or numbered list item."""
style = "List Bullet" if not ordered else "List Number"
para = doc.add_paragraph(style=style)
if level > 0:
para.paragraph_format.left_indent = Inches(0.25 * (level + 1))
_parse_inline(para, text)
for run in para.runs:
run.font.size = Pt(10)
def _parse_md_table(raw_rows: list[str]) -> list[list[str]]:
"""Convert raw Markdown table lines to a list of cell lists."""
result = []
for line in raw_rows:
# Skip separator rows (---|---)
if re.match(r"^\s*\|?[\s\-:]+\|[\s\-:|]+\s*$", line):
continue
cells = [c.strip() for c in line.strip().strip("|").split("|")]
if cells:
result.append(cells)
return result
# ── Main converter ────────────────────────────────────────────────────────────
def convert(md_path: Path, docx_path: Path) -> None:
doc = Document()
# Page margins
for section in doc.sections:
section.top_margin = Inches(1.0)
section.bottom_margin = Inches(1.0)
section.left_margin = Inches(1.2)
section.right_margin = Inches(1.2)
# Default body style
style = doc.styles["Normal"]
style.font.name = "Calibri"
style.font.size = Pt(10.5)
style.font.color.rgb = DARK_GREY
lines = md_path.read_text(encoding="utf-8").splitlines()
i = 0
in_code_block = False
code_lines: list[str] = []
table_rows: list[str] = []
in_table = False
while i < len(lines):
line = lines[i]
# ── Fenced code block ──────────────────────────────────────────────
if line.strip().startswith("```"):
if not in_code_block:
in_code_block = True
code_lines = []
else:
in_code_block = False
_add_code_block(doc, code_lines)
i += 1
continue
if in_code_block:
code_lines.append(line)
i += 1
continue
# ── Table detection ────────────────────────────────────────────────
is_table_line = "|" in line and line.strip().startswith("|")
if is_table_line:
table_rows.append(line)
i += 1
continue
elif table_rows:
parsed = _parse_md_table(table_rows)
if parsed:
_add_table(doc, parsed)
table_rows = []
# ── Headings ────────────────────────────────────────────────────────
m = re.match(r"^(#{1,4})\s+(.+)$", line)
if m:
level = len(m.group(1))
_add_heading(doc, m.group(2), level)
i += 1
continue
# ── Horizontal rule ─────────────────────────────────────────────────
if re.match(r"^[-*_]{3,}\s*$", line.strip()):
para = doc.add_paragraph()
pPr = para._p.get_or_add_pPr()
pBdr = OxmlElement("w:pBdr")
bottom = OxmlElement("w:bottom")
bottom.set(qn("w:val"), "single")
bottom.set(qn("w:sz"), "6")
bottom.set(qn("w:space"), "1")
bottom.set(qn("w:color"), "2E74B5")
pBdr.append(bottom)
pPr.append(pBdr)
i += 1
continue
# ── Unordered list ──────────────────────────────────────────────────
m = re.match(r"^(\s*)[-*]\s+(.+)$", line)
if m:
indent = len(m.group(1)) // 2
_add_list_item(doc, m.group(2), indent, ordered=False, counter=0)
i += 1
continue
# ── Ordered list ────────────────────────────────────────────────────
m = re.match(r"^(\s*)\d+\.\s+(.+)$", line)
if m:
indent = len(m.group(1)) // 2
_add_list_item(doc, m.group(2), indent, ordered=True, counter=0)
i += 1
continue
# ── Blank line ──────────────────────────────────────────────────────
if not line.strip():
i += 1
continue
# ── Plain paragraph ─────────────────────────────────────────────────
para = doc.add_paragraph()
para.paragraph_format.space_after = Pt(4)
_parse_inline(para, line)
for run in para.runs:
run.font.size = Pt(10.5)
i += 1
# Flush any remaining table
if table_rows:
parsed = _parse_md_table(table_rows)
if parsed:
_add_table(doc, parsed)
doc.save(str(docx_path))
print(f" Written: {docx_path} ({docx_path.stat().st_size // 1024} KB)")
# ── Entry point ───────────────────────────────────────────────────────────────
if __name__ == "__main__":
docs_dir = Path(__file__).parent
files = [
("HLD.md", "HLD.docx"),
("LLD.md", "LLD.docx"),
("MANUAL.md", "MANUAL.docx"),
]
print("Converting Markdown → Word (.docx) ...")
for md_name, docx_name in files:
md_path = docs_dir / md_name
docx_path = docs_dir / docx_name
print(f" Processing {md_name} ...")
convert(md_path, docx_path)
print("Done.")