chore: add project agent skills
This commit is contained in:
@@ -0,0 +1,146 @@
|
||||
---
|
||||
name: devops-engineer
|
||||
description: Creates Dockerfiles, configures CI/CD pipelines, writes Kubernetes manifests, and generates Terraform/Pulumi infrastructure templates. Handles deployment automation, GitOps configuration, incident response runbooks, and internal developer platform tooling. Use when setting up CI/CD pipelines, containerizing applications, managing infrastructure as code, deploying to Kubernetes clusters, configuring cloud platforms, automating releases, or responding to production incidents. Invoke for pipelines, Docker, Kubernetes, GitOps, Terraform, GitHub Actions, on-call, or platform engineering.
|
||||
license: MIT
|
||||
metadata:
|
||||
author: https://github.com/Jeffallan
|
||||
version: "1.1.1"
|
||||
domain: devops
|
||||
triggers: DevOps, CI/CD, deployment, Docker, Kubernetes, Terraform, GitHub Actions, infrastructure, platform engineering, incident response, on-call, self-service
|
||||
role: engineer
|
||||
scope: implementation
|
||||
output-format: code
|
||||
related-skills: terraform-engineer, kubernetes-specialist, sre-engineer, monitoring-expert, security-reviewer
|
||||
---
|
||||
|
||||
# DevOps Engineer
|
||||
|
||||
Senior DevOps engineer specializing in CI/CD pipelines, infrastructure as code, and deployment automation.
|
||||
|
||||
## Role Definition
|
||||
|
||||
You are a senior DevOps engineer with 10+ years of experience. You operate with three perspectives:
|
||||
- **Build Hat**: Automating build, test, and packaging
|
||||
- **Deploy Hat**: Orchestrating deployments across environments
|
||||
- **Ops Hat**: Ensuring reliability, monitoring, and incident response
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
- Setting up CI/CD pipelines (GitHub Actions, GitLab CI, Jenkins)
|
||||
- Containerizing applications (Docker, Docker Compose)
|
||||
- Kubernetes deployments and configurations
|
||||
- Infrastructure as code (Terraform, Pulumi)
|
||||
- Cloud platform configuration (AWS, GCP, Azure)
|
||||
- Deployment strategies (blue-green, canary, rolling)
|
||||
- Building internal developer platforms and self-service tools
|
||||
- Incident response, on-call, and production troubleshooting
|
||||
- Release automation and artifact management
|
||||
|
||||
## Core Workflow
|
||||
|
||||
1. **Assess** - Understand application, environments, requirements
|
||||
2. **Design** - Pipeline structure, deployment strategy
|
||||
3. **Implement** - IaC, Dockerfiles, CI/CD configs
|
||||
4. **Validate** - Run `terraform plan`, lint configs, execute unit/integration tests; confirm no destructive changes before proceeding
|
||||
5. **Deploy** - Roll out with verification; run smoke tests post-deployment
|
||||
6. **Monitor** - Set up observability, alerts; confirm rollback procedure is ready before going live
|
||||
|
||||
## Reference Guide
|
||||
|
||||
Load detailed guidance based on context:
|
||||
|
||||
| Topic | Reference | Load When |
|
||||
|-------|-----------|-----------|
|
||||
| GitHub Actions | `references/github-actions.md` | Setting up CI/CD pipelines, GitHub workflows |
|
||||
| Docker | `references/docker-patterns.md` | Containerizing applications, writing Dockerfiles |
|
||||
| Kubernetes | `references/kubernetes.md` | K8s deployments, services, ingress, pods |
|
||||
| Terraform | `references/terraform-iac.md` | Infrastructure as code, AWS/GCP provisioning |
|
||||
| Deployment | `references/deployment-strategies.md` | Blue-green, canary, rolling updates, rollback |
|
||||
| Platform | `references/platform-engineering.md` | Self-service infra, developer portals, golden paths, Backstage |
|
||||
| Release | `references/release-automation.md` | Artifact management, feature flags, multi-platform CI/CD |
|
||||
| Incidents | `references/incident-response.md` | Production outages, on-call, MTTR, postmortems, runbooks |
|
||||
|
||||
## Constraints
|
||||
|
||||
### MUST DO
|
||||
- Use infrastructure as code (never manual changes)
|
||||
- Implement health checks and readiness probes
|
||||
- Store secrets in secret managers (not env files)
|
||||
- Enable container scanning in CI/CD
|
||||
- Document rollback procedures
|
||||
- Use GitOps for Kubernetes (ArgoCD, Flux)
|
||||
|
||||
### MUST NOT DO
|
||||
- Deploy to production without explicit approval
|
||||
- Store secrets in code or CI/CD variables
|
||||
- Skip staging environment testing
|
||||
- Ignore resource limits in containers
|
||||
- Use `latest` tag in production
|
||||
- Deploy on Fridays without monitoring
|
||||
|
||||
## Output Templates
|
||||
|
||||
Provide: CI/CD pipeline config, Dockerfile, K8s/Terraform files, deployment verification, rollback procedure
|
||||
|
||||
### Minimal GitHub Actions Example
|
||||
|
||||
```yaml
|
||||
name: CI
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
jobs:
|
||||
build-test-push:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Build image
|
||||
run: docker build -t myapp:${{ github.sha }} .
|
||||
- name: Run tests
|
||||
run: docker run --rm myapp:${{ github.sha }} pytest
|
||||
- name: Scan image
|
||||
uses: aquasecurity/trivy-action@master
|
||||
with:
|
||||
image-ref: myapp:${{ github.sha }}
|
||||
- name: Push to registry
|
||||
run: |
|
||||
docker tag myapp:${{ github.sha }} ghcr.io/org/myapp:${{ github.sha }}
|
||||
docker push ghcr.io/org/myapp:${{ github.sha }}
|
||||
```
|
||||
|
||||
### Minimal Dockerfile Example
|
||||
|
||||
```dockerfile
|
||||
FROM python:3.12-slim AS builder
|
||||
WORKDIR /app
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
FROM python:3.12-slim
|
||||
WORKDIR /app
|
||||
COPY --from=builder /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packages
|
||||
COPY . .
|
||||
USER nonroot
|
||||
HEALTHCHECK --interval=30s --timeout=5s CMD curl -f http://localhost:8080/health || exit 1
|
||||
CMD ["python", "main.py"]
|
||||
```
|
||||
|
||||
### Rollback Procedure Example
|
||||
|
||||
```bash
|
||||
# Kubernetes: roll back to previous deployment revision
|
||||
kubectl rollout undo deployment/myapp -n production
|
||||
kubectl rollout status deployment/myapp -n production
|
||||
|
||||
# Verify rollback succeeded
|
||||
kubectl get pods -n production -l app=myapp
|
||||
curl -f https://myapp.example.com/health
|
||||
```
|
||||
|
||||
Always document the rollback command and verification step in the PR or change ticket before deploying.
|
||||
|
||||
## Knowledge Reference
|
||||
|
||||
GitHub Actions, GitLab CI, Jenkins, CircleCI, Docker, Kubernetes, Helm, ArgoCD, Flux, Terraform, Pulumi, Crossplane, AWS/GCP/Azure, Prometheus, Grafana, PagerDuty, Backstage, LaunchDarkly, Flagger
|
||||
|
||||
[Documentation](https://jeffallan.github.io/claude-skills/skills/devops/devops-engineer/)
|
||||
@@ -0,0 +1,241 @@
|
||||
# Deployment Strategies
|
||||
|
||||
## Strategy Comparison
|
||||
|
||||
| Strategy | Use When | Rollback | Risk |
|
||||
|----------|----------|----------|------|
|
||||
| **Rolling** | Standard updates, can tolerate mixed versions | Automatic via health checks | Low |
|
||||
| **Blue-Green** | Zero downtime, instant rollback needed | Switch traffic to old env | Medium |
|
||||
| **Canary** | Risk mitigation, gradual rollout | Scale down canary | Low |
|
||||
| **Recreate** | Stateful apps, breaking changes | Redeploy previous version | High |
|
||||
|
||||
## Rolling Deployment (Kubernetes)
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
spec:
|
||||
strategy:
|
||||
type: RollingUpdate
|
||||
rollingUpdate:
|
||||
maxSurge: 25% # Max pods above desired
|
||||
maxUnavailable: 25% # Max pods unavailable
|
||||
```
|
||||
|
||||
## Blue-Green with Ingress
|
||||
|
||||
```yaml
|
||||
# Blue deployment (current)
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: app-blue
|
||||
labels:
|
||||
version: blue
|
||||
---
|
||||
# Green deployment (new)
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: app-green
|
||||
labels:
|
||||
version: green
|
||||
---
|
||||
# Service pointing to active version
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: app
|
||||
spec:
|
||||
selector:
|
||||
version: blue # Switch to 'green' for cutover
|
||||
```
|
||||
|
||||
## Canary with Istio
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.istio.io/v1beta1
|
||||
kind: VirtualService
|
||||
metadata:
|
||||
name: app
|
||||
spec:
|
||||
hosts:
|
||||
- app
|
||||
http:
|
||||
- match:
|
||||
- headers:
|
||||
canary:
|
||||
exact: "true"
|
||||
route:
|
||||
- destination:
|
||||
host: app-canary
|
||||
- route:
|
||||
- destination:
|
||||
host: app-stable
|
||||
weight: 90
|
||||
- destination:
|
||||
host: app-canary
|
||||
weight: 10
|
||||
```
|
||||
|
||||
## Rollback Procedures
|
||||
|
||||
### Kubernetes Rollback
|
||||
```bash
|
||||
# View rollout history
|
||||
kubectl rollout history deployment/app
|
||||
|
||||
# Rollback to previous
|
||||
kubectl rollout undo deployment/app
|
||||
|
||||
# Rollback to specific revision
|
||||
kubectl rollout undo deployment/app --to-revision=2
|
||||
|
||||
# Check status
|
||||
kubectl rollout status deployment/app
|
||||
```
|
||||
|
||||
### ArgoCD Rollback
|
||||
```bash
|
||||
argocd app rollback app-prod --revision=123
|
||||
```
|
||||
|
||||
### Terraform Rollback
|
||||
```bash
|
||||
# Identify previous state
|
||||
terraform state list
|
||||
|
||||
# Import previous configuration
|
||||
git checkout HEAD~1 -- main.tf
|
||||
terraform apply
|
||||
```
|
||||
|
||||
## Pre-deployment Checklist
|
||||
|
||||
- [ ] Database migrations are backward compatible
|
||||
- [ ] Feature flags for new functionality
|
||||
- [ ] Monitoring dashboards updated
|
||||
- [ ] Alert thresholds reviewed
|
||||
- [ ] Rollback procedure documented
|
||||
- [ ] Staging tested and approved
|
||||
- [ ] Team notified of deployment window
|
||||
|
||||
## Post-deployment Verification
|
||||
|
||||
```bash
|
||||
# Check pod status
|
||||
kubectl get pods -l app=app
|
||||
|
||||
# Check logs for errors
|
||||
kubectl logs -l app=app --tail=100 | grep -i error
|
||||
|
||||
# Verify endpoints
|
||||
curl -f https://app.example.com/health
|
||||
|
||||
# Check metrics
|
||||
# - Error rate < 1%
|
||||
# - Latency p99 < 500ms
|
||||
# - No memory/CPU spikes
|
||||
```
|
||||
|
||||
## Deployment Metrics (DORA)
|
||||
|
||||
Track four key metrics:
|
||||
- **Deployment Frequency**: Target 10+/day
|
||||
- **Lead Time for Changes**: Target <1 hour
|
||||
- **Change Failure Rate**: Target <5%
|
||||
- **MTTR**: Target <30 minutes
|
||||
|
||||
```yaml
|
||||
# Prometheus metrics for DORA tracking
|
||||
- record: deployment:frequency:1d
|
||||
expr: count_over_time(deployment_completed[1d])
|
||||
|
||||
- record: deployment:lead_time:p95
|
||||
expr: histogram_quantile(0.95,
|
||||
rate(commit_to_deploy_seconds_bucket[1h]))
|
||||
|
||||
- record: deployment:failure_rate
|
||||
expr: |
|
||||
sum(rate(deployment_failed[1h]))
|
||||
/ sum(rate(deployment_total[1h]))
|
||||
```
|
||||
|
||||
## Advanced Canary with Automated Analysis
|
||||
|
||||
```yaml
|
||||
# Flagger: Automated canary with rollback
|
||||
apiVersion: flagger.app/v1beta1
|
||||
kind: Canary
|
||||
metadata:
|
||||
name: api
|
||||
spec:
|
||||
provider: istio
|
||||
targetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: api
|
||||
progressDeadlineSeconds: 60
|
||||
service:
|
||||
port: 8080
|
||||
trafficPolicy:
|
||||
tls:
|
||||
mode: ISTIO_MUTUAL
|
||||
analysis:
|
||||
interval: 30s
|
||||
threshold: 5
|
||||
maxWeight: 50
|
||||
stepWeight: 10
|
||||
metrics:
|
||||
- name: error-rate
|
||||
templateRef:
|
||||
name: error-rate
|
||||
thresholdRange:
|
||||
max: 1
|
||||
- name: latency
|
||||
templateRef:
|
||||
name: latency
|
||||
thresholdRange:
|
||||
max: 500
|
||||
webhooks:
|
||||
- name: acceptance-test
|
||||
type: pre-rollout
|
||||
url: http://test-runner/
|
||||
- name: load-test
|
||||
url: http://loadtester/
|
||||
timeout: 5s
|
||||
metadata:
|
||||
type: bash
|
||||
cmd: "hey -z 1m -q 10 http://api-canary:8080/"
|
||||
```
|
||||
|
||||
## Shadow Deployment
|
||||
|
||||
```yaml
|
||||
# Mirror traffic to shadow deployment
|
||||
apiVersion: networking.istio.io/v1beta1
|
||||
kind: VirtualService
|
||||
metadata:
|
||||
name: api
|
||||
spec:
|
||||
hosts:
|
||||
- api
|
||||
http:
|
||||
- match:
|
||||
- headers:
|
||||
x-test-version:
|
||||
exact: "v2"
|
||||
route:
|
||||
- destination:
|
||||
host: api
|
||||
subset: v2
|
||||
mirror:
|
||||
host: api
|
||||
subset: v2-shadow
|
||||
mirrorPercentage:
|
||||
value: 100
|
||||
- route:
|
||||
- destination:
|
||||
host: api
|
||||
subset: v1
|
||||
```
|
||||
@@ -0,0 +1,113 @@
|
||||
# Docker Patterns
|
||||
|
||||
## Multi-stage Dockerfile (Node.js)
|
||||
|
||||
```dockerfile
|
||||
# Build stage
|
||||
FROM node:20-alpine AS builder
|
||||
WORKDIR /app
|
||||
COPY package*.json ./
|
||||
RUN npm ci --only=production && npm cache clean --force
|
||||
COPY . .
|
||||
RUN npm run build
|
||||
|
||||
# Production stage
|
||||
FROM node:20-alpine AS runner
|
||||
WORKDIR /app
|
||||
ENV NODE_ENV=production
|
||||
RUN addgroup -g 1001 -S nodejs && adduser -S nodejs -u 1001
|
||||
COPY --from=builder --chown=nodejs:nodejs /app/dist ./dist
|
||||
COPY --from=builder --chown=nodejs:nodejs /app/node_modules ./node_modules
|
||||
USER nodejs
|
||||
EXPOSE 3000
|
||||
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s \
|
||||
CMD wget -qO- http://localhost:3000/health || exit 1
|
||||
CMD ["node", "dist/main.js"]
|
||||
```
|
||||
|
||||
## Multi-stage Dockerfile (Python)
|
||||
|
||||
```dockerfile
|
||||
FROM python:3.12-slim AS builder
|
||||
WORKDIR /app
|
||||
RUN pip install --no-cache-dir poetry
|
||||
COPY pyproject.toml poetry.lock ./
|
||||
RUN poetry export -f requirements.txt --output requirements.txt
|
||||
RUN pip wheel --no-cache-dir --wheel-dir /wheels -r requirements.txt
|
||||
|
||||
FROM python:3.12-slim AS runner
|
||||
WORKDIR /app
|
||||
RUN useradd -m -u 1001 appuser
|
||||
COPY --from=builder /wheels /wheels
|
||||
RUN pip install --no-cache-dir /wheels/*
|
||||
COPY --chown=appuser:appuser . .
|
||||
USER appuser
|
||||
EXPOSE 8000
|
||||
HEALTHCHECK --interval=30s --timeout=3s \
|
||||
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"
|
||||
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
```
|
||||
|
||||
## Docker Compose (Development)
|
||||
|
||||
```yaml
|
||||
version: '3.8'
|
||||
services:
|
||||
app:
|
||||
build:
|
||||
context: .
|
||||
target: builder # Use dev stage
|
||||
volumes:
|
||||
- .:/app
|
||||
- /app/node_modules
|
||||
ports:
|
||||
- "3000:3000"
|
||||
environment:
|
||||
- DATABASE_URL=postgres://user:pass@db:5432/app
|
||||
depends_on:
|
||||
db:
|
||||
condition: service_healthy
|
||||
|
||||
db:
|
||||
image: postgres:16-alpine
|
||||
environment:
|
||||
POSTGRES_USER: user
|
||||
POSTGRES_PASSWORD: pass
|
||||
POSTGRES_DB: app
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U user -d app"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
```
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
| Practice | Implementation |
|
||||
|----------|----------------|
|
||||
| Non-root user | `USER nodejs` or `USER 1001` |
|
||||
| Minimal base image | Use `-alpine` or `-slim` variants |
|
||||
| No secrets in image | Use runtime env vars or secrets |
|
||||
| Pin versions | `FROM node:20.10.0-alpine` not `latest` |
|
||||
| Scan images | `docker scout`, `trivy`, `snyk` |
|
||||
| Health checks | `HEALTHCHECK` instruction |
|
||||
| .dockerignore | Exclude `node_modules`, `.git`, `.env` |
|
||||
|
||||
## .dockerignore Template
|
||||
|
||||
```
|
||||
node_modules
|
||||
.git
|
||||
.env*
|
||||
*.md
|
||||
Dockerfile*
|
||||
docker-compose*
|
||||
.dockerignore
|
||||
coverage
|
||||
.nyc_output
|
||||
```
|
||||
@@ -0,0 +1,139 @@
|
||||
# GitHub Actions Pipelines
|
||||
|
||||
## Complete CI/CD Pipeline
|
||||
|
||||
```yaml
|
||||
name: CI/CD Pipeline
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main, develop]
|
||||
pull_request:
|
||||
branches: [main]
|
||||
|
||||
env:
|
||||
REGISTRY: ghcr.io
|
||||
IMAGE_NAME: ${{ github.repository }}
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
- run: npm ci
|
||||
- run: npm test
|
||||
- run: npm run lint
|
||||
|
||||
build:
|
||||
needs: test
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
outputs:
|
||||
image-tag: ${{ steps.meta.outputs.tags }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: docker/setup-buildx-action@v3
|
||||
- uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ${{ env.REGISTRY }}
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
- id: meta
|
||||
uses: docker/metadata-action@v5
|
||||
with:
|
||||
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
|
||||
tags: |
|
||||
type=sha,prefix=
|
||||
type=ref,event=branch
|
||||
- uses: docker/build-push-action@v5
|
||||
with:
|
||||
context: .
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
deploy-staging:
|
||||
needs: build
|
||||
if: github.ref == 'refs/heads/develop'
|
||||
runs-on: ubuntu-latest
|
||||
environment: staging
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- run: |
|
||||
kubectl set image deployment/app app=${{ needs.build.outputs.image-tag }}
|
||||
|
||||
deploy-production:
|
||||
needs: build
|
||||
if: github.ref == 'refs/heads/main'
|
||||
runs-on: ubuntu-latest
|
||||
environment: production
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- run: |
|
||||
kubectl set image deployment/app app=${{ needs.build.outputs.image-tag }}
|
||||
```
|
||||
|
||||
## Common Workflow Patterns
|
||||
|
||||
### Matrix Builds (Multi-version testing)
|
||||
```yaml
|
||||
jobs:
|
||||
test:
|
||||
strategy:
|
||||
matrix:
|
||||
node-version: [18, 20, 22]
|
||||
os: [ubuntu-latest, macos-latest]
|
||||
runs-on: ${{ matrix.os }}
|
||||
steps:
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: ${{ matrix.node-version }}
|
||||
```
|
||||
|
||||
### Reusable Workflows
|
||||
```yaml
|
||||
# .github/workflows/deploy.yml
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
environment:
|
||||
required: true
|
||||
type: string
|
||||
secrets:
|
||||
DEPLOY_KEY:
|
||||
required: true
|
||||
|
||||
jobs:
|
||||
deploy:
|
||||
runs-on: ubuntu-latest
|
||||
environment: ${{ inputs.environment }}
|
||||
steps:
|
||||
- run: echo "Deploying to ${{ inputs.environment }}"
|
||||
```
|
||||
|
||||
### Caching Dependencies
|
||||
```yaml
|
||||
- uses: actions/cache@v4
|
||||
with:
|
||||
path: ~/.npm
|
||||
key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-node-
|
||||
```
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Action | Purpose |
|
||||
|--------|---------|
|
||||
| `actions/checkout@v4` | Clone repository |
|
||||
| `actions/setup-node@v4` | Install Node.js |
|
||||
| `docker/build-push-action@v5` | Build and push Docker image |
|
||||
| `docker/metadata-action@v5` | Generate Docker tags |
|
||||
| `actions/cache@v4` | Cache dependencies |
|
||||
@@ -0,0 +1,331 @@
|
||||
# Incident Response
|
||||
|
||||
## Response Metrics
|
||||
|
||||
- **MTTD** (Mean Time to Detect): Target < 5 minutes
|
||||
- **MTTA** (Mean Time to Acknowledge): Target < 5 minutes
|
||||
- **MTTR** (Mean Time to Resolve): Target < 30 minutes
|
||||
- **MTBF** (Mean Time Between Failures): Maximize
|
||||
|
||||
### Severity Levels
|
||||
|
||||
| Level | Impact | Response | Example |
|
||||
|-------|--------|----------|---------|
|
||||
| SEV1 | Complete outage | Immediate | Database down, payment failed |
|
||||
| SEV2 | Major degradation | 15 min | API latency >5s, 50% errors |
|
||||
| SEV3 | Minor degradation | 1 hour | Non-critical feature broken |
|
||||
| SEV4 | Low impact | Business hours | UI glitch, logging issues |
|
||||
|
||||
## Runbook Template
|
||||
|
||||
```markdown
|
||||
# Runbook: High API Error Rate
|
||||
|
||||
## Symptoms
|
||||
- Alert: `api_error_rate > 0.05`
|
||||
- Dashboard: https://grafana.example.com/d/api-errors
|
||||
|
||||
## Impact
|
||||
Users cannot complete purchases (~$X per minute)
|
||||
|
||||
## Triage
|
||||
1. Check dashboard for affected endpoints
|
||||
2. Check recent deployments: `kubectl rollout history deployment/api`
|
||||
3. Check dependencies: database, redis, external APIs
|
||||
|
||||
## Resolution
|
||||
|
||||
### Option 1: Rollback
|
||||
kubectl rollout undo deployment/api -n production
|
||||
|
||||
### Option 2: Scale Up
|
||||
kubectl scale deployment/api --replicas=10 -n production
|
||||
|
||||
### Option 3: Fix Config
|
||||
kubectl set env deployment/api DB_POOL_SIZE=50 -n production
|
||||
|
||||
## Verification
|
||||
- [ ] Error rate <1%
|
||||
- [ ] P95 latency <500ms
|
||||
- [ ] Health checks passing
|
||||
|
||||
## Communication
|
||||
- Update status page
|
||||
- Notify #incidents
|
||||
- Post if user-facing
|
||||
```
|
||||
|
||||
## Auto-Remediation Script
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
import kubernetes, prometheus_api_client
|
||||
|
||||
class IncidentRemediator:
|
||||
def check_high_error_rate(self):
|
||||
query = 'rate(http_requests_total{status=~"5.."}[5m]) > 0.05'
|
||||
result = self.prometheus.custom_query(query)
|
||||
return len(result) > 0
|
||||
|
||||
def rollback_deployment(self, namespace, deployment):
|
||||
body = {'spec': {'rollbackTo': {'revision': 0}}}
|
||||
self.k8s.patch_namespaced_deployment(deployment, namespace, body)
|
||||
|
||||
def remediate(self):
|
||||
if self.check_high_error_rate():
|
||||
if self.rollback_deployment('production', 'api'):
|
||||
time.sleep(120)
|
||||
if not self.check_high_error_rate():
|
||||
return # Success
|
||||
# Escalate if remediation fails
|
||||
self.create_incident("Auto-remediation failed")
|
||||
```
|
||||
|
||||
## Postmortem Template
|
||||
|
||||
```markdown
|
||||
# Postmortem: API Outage - 2024-01-15
|
||||
|
||||
**Date**: January 15, 2024
|
||||
**Duration**: 45 minutes (14:23 - 15:08 UTC)
|
||||
**Severity**: SEV1
|
||||
**Impact**: Complete API outage, ~$25K revenue loss
|
||||
|
||||
## Summary
|
||||
API became unresponsive due to database connection pool exhaustion
|
||||
from slow query in v2.3.1.
|
||||
|
||||
## Timeline (UTC)
|
||||
- 14:23 - Alert fired
|
||||
- 14:27 - Incident declared SEV1
|
||||
- 14:30 - Rollback initiated
|
||||
- 14:45 - Identified slow query
|
||||
- 14:50 - Killed queries
|
||||
- 15:08 - Resolved
|
||||
|
||||
## Root Cause
|
||||
New query missing index on `user_id`, causing full table scans that
|
||||
exhausted connection pool under load.
|
||||
|
||||
## Impact
|
||||
- 100% API failure for 45 minutes
|
||||
- 15,000 users affected
|
||||
- $25K revenue loss
|
||||
- 200+ support tickets
|
||||
|
||||
## Action Items
|
||||
| Action | Owner | Deadline |
|
||||
|--------|-------|----------|
|
||||
| Add index on user_id | DB team | 2024-01-16 |
|
||||
| Add query perf testing | Platform | 2024-01-22 |
|
||||
| Increase staging DB size | Infra | 2024-01-30 |
|
||||
|
||||
## Lessons Learned
|
||||
- Performance testing must use production-scale data
|
||||
- Connection pool exhaustion needs active intervention
|
||||
- Consider circuit breakers for DB operations
|
||||
```
|
||||
|
||||
## PagerDuty Configuration
|
||||
|
||||
```yaml
|
||||
schedules:
|
||||
- name: Primary On-Call
|
||||
time_zone: America/New_York
|
||||
layers:
|
||||
- rotation_turn_length_seconds: 604800 # 1 week
|
||||
users: [PXXXXXX, PXXXXXX, PXXXXXX]
|
||||
|
||||
escalation_policies:
|
||||
- name: Production
|
||||
rules:
|
||||
- escalation_delay_in_minutes: 0
|
||||
targets: [{type: schedule, id: primary}]
|
||||
- escalation_delay_in_minutes: 15
|
||||
targets: [{type: schedule, id: secondary}]
|
||||
- escalation_delay_in_minutes: 30
|
||||
targets: [{type: user, id: manager}]
|
||||
```
|
||||
|
||||
## Chaos Engineering
|
||||
|
||||
```yaml
|
||||
# chaos-mesh: Pod failure test
|
||||
apiVersion: chaos-mesh.org/v1alpha1
|
||||
kind: PodChaos
|
||||
metadata:
|
||||
name: pod-failure-test
|
||||
spec:
|
||||
action: pod-failure
|
||||
mode: one
|
||||
duration: "30s"
|
||||
selector:
|
||||
namespaces: [production]
|
||||
labelSelectors:
|
||||
app: api
|
||||
scheduler:
|
||||
cron: "@every 2h"
|
||||
```
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Game Day: Database failover drill
|
||||
|
||||
echo "🎮 Game Day: Database failover"
|
||||
slack-cli -d incidents "Starting failover drill"
|
||||
|
||||
# Simulate failure
|
||||
kubectl delete pod postgres-0 -n production
|
||||
|
||||
# Monitor recovery
|
||||
start=$(date +%s)
|
||||
while ! kubectl get pod postgres-1 | grep Running; do
|
||||
sleep 5
|
||||
done
|
||||
duration=$(($(date +%s) - start))
|
||||
|
||||
echo "Failover: ${duration}s" >> results.md
|
||||
curl -f https://api.example.com/health || echo "FAIL"
|
||||
```
|
||||
|
||||
## Evidence Collection & Forensics
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# collect-evidence.sh - Preserve incident evidence
|
||||
|
||||
INCIDENT_ID=$1
|
||||
EVIDENCE_DIR="incidents/${INCIDENT_ID}/evidence"
|
||||
mkdir -p $EVIDENCE_DIR
|
||||
|
||||
# Preserve logs
|
||||
kubectl logs -l app=api --all-containers --timestamps \
|
||||
--since=2h > $EVIDENCE_DIR/pod-logs.txt
|
||||
|
||||
# Capture current state
|
||||
kubectl get all -n production -o yaml > $EVIDENCE_DIR/k8s-state.yaml
|
||||
kubectl describe pods -n production > $EVIDENCE_DIR/pod-details.txt
|
||||
|
||||
# Network traces
|
||||
kubectl exec -n production deploy/api -- \
|
||||
tcpdump -i any -w /tmp/capture.pcap -G 60 -W 5 &
|
||||
|
||||
# Memory/CPU snapshot
|
||||
kubectl top pods -n production > $EVIDENCE_DIR/resource-usage.txt
|
||||
|
||||
# Git commit at time of incident
|
||||
git log --since="2 hours ago" --oneline > $EVIDENCE_DIR/recent-commits.txt
|
||||
|
||||
# Database queries
|
||||
psql -c "SELECT * FROM pg_stat_activity" > $EVIDENCE_DIR/db-activity.txt
|
||||
|
||||
# Create timeline
|
||||
echo "$(date): Evidence collection completed" >> $EVIDENCE_DIR/timeline.txt
|
||||
```
|
||||
|
||||
## Communication Templates
|
||||
|
||||
```markdown
|
||||
## SEV1 Initial Notification
|
||||
|
||||
**INCIDENT ALERT - SEV1**
|
||||
|
||||
**Status**: Investigating
|
||||
**Impact**: Payment API unavailable (100% error rate)
|
||||
**Started**: 2024-01-15 14:23 UTC
|
||||
**Affected**: All users (~15K active sessions)
|
||||
**Lead**: @oncall-engineer
|
||||
**War Room**: https://zoom.us/incident-123
|
||||
|
||||
Updates every 15 minutes or on major change.
|
||||
|
||||
---
|
||||
|
||||
## SEV1 Resolution Notification
|
||||
|
||||
**INCIDENT RESOLVED**
|
||||
|
||||
**Summary**: Payment API restored after database connection pool exhaustion
|
||||
**Duration**: 45 minutes (14:23 - 15:08 UTC)
|
||||
**Resolution**: Rollback to v2.3.0 + query optimization
|
||||
**Impact**: 15K users, ~$25K revenue loss
|
||||
**Next Steps**: Postmortem scheduled for Jan 16 10am
|
||||
|
||||
Thanks to @oncall-team for rapid response.
|
||||
```
|
||||
|
||||
## Incident Classification
|
||||
|
||||
| Type | Examples | Response Team | Escalation |
|
||||
|------|----------|---------------|------------|
|
||||
| **Security** | Breach, data leak, unauthorized access | Security + DevOps | CISO, Legal |
|
||||
| **Service** | Outage, degradation, errors | DevOps + SRE | Engineering VP |
|
||||
| **Data** | Corruption, loss, sync issues | DBA + DevOps | CTO |
|
||||
| **Compliance** | GDPR, SOC2, audit violations | Compliance + Legal | CEO |
|
||||
| **Third-party** | Provider outage, API failures | DevOps + Product | Account manager |
|
||||
|
||||
## Security Incident Specifics
|
||||
|
||||
```bash
|
||||
# Compromise investigation checklist
|
||||
□ Isolate affected systems
|
||||
□ Preserve logs and memory dumps
|
||||
□ Identify attack vector
|
||||
□ Check for lateral movement
|
||||
□ Scan for malware/backdoors
|
||||
□ Review access logs for 30 days
|
||||
□ Identify data accessed
|
||||
□ Assess exfiltration risk
|
||||
□ Check for persistence mechanisms
|
||||
□ Coordinate with security team
|
||||
□ Notify legal if PII involved
|
||||
□ Document chain of custody
|
||||
```
|
||||
|
||||
## Compliance Requirements
|
||||
|
||||
```yaml
|
||||
# Incident notification requirements
|
||||
gdpr:
|
||||
notification_deadline: 72h
|
||||
authority: Data Protection Officer
|
||||
required_info:
|
||||
- Nature of breach
|
||||
- Data categories affected
|
||||
- Number of individuals
|
||||
- Consequences
|
||||
- Remediation measures
|
||||
|
||||
sox:
|
||||
notification_deadline: immediate
|
||||
authority: Audit Committee
|
||||
documentation:
|
||||
- Financial impact
|
||||
- Control failures
|
||||
- Remediation plan
|
||||
|
||||
pci_dss:
|
||||
notification_deadline: 24h
|
||||
authority: Card brands + acquirer
|
||||
required_info:
|
||||
- Cardholder data affected
|
||||
- Incident timeline
|
||||
- Forensic investigation
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Maintain runbooks for critical services
|
||||
- Practice with game days monthly
|
||||
- Automate common remediation
|
||||
- Keep postmortems blameless
|
||||
- Track incident metrics
|
||||
- Test recovery procedures
|
||||
- Document all incidents
|
||||
- Improve detection continuously
|
||||
- Preserve evidence chain properly
|
||||
- Coordinate communication clearly
|
||||
- Escalate security incidents immediately
|
||||
- Understand compliance obligations
|
||||
- Train team on response procedures
|
||||
- Review and update playbooks quarterly
|
||||
@@ -0,0 +1,154 @@
|
||||
# Kubernetes Manifests
|
||||
|
||||
## Complete Deployment Stack
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: app
|
||||
labels:
|
||||
app: app
|
||||
spec:
|
||||
replicas: 3
|
||||
selector:
|
||||
matchLabels:
|
||||
app: app
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: app
|
||||
spec:
|
||||
containers:
|
||||
- name: app
|
||||
image: ghcr.io/org/app:latest
|
||||
ports:
|
||||
- containerPort: 3000
|
||||
resources:
|
||||
requests:
|
||||
memory: "128Mi"
|
||||
cpu: "100m"
|
||||
limits:
|
||||
memory: "256Mi"
|
||||
cpu: "500m"
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 3000
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 10
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ready
|
||||
port: 3000
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
env:
|
||||
- name: DATABASE_URL
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: app-secrets
|
||||
key: database-url
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: app
|
||||
spec:
|
||||
selector:
|
||||
app: app
|
||||
ports:
|
||||
- port: 80
|
||||
targetPort: 3000
|
||||
type: ClusterIP
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: app
|
||||
annotations:
|
||||
cert-manager.io/cluster-issuer: letsencrypt-prod
|
||||
spec:
|
||||
ingressClassName: nginx
|
||||
tls:
|
||||
- hosts: [app.example.com]
|
||||
secretName: app-tls
|
||||
rules:
|
||||
- host: app.example.com
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: app
|
||||
port:
|
||||
number: 80
|
||||
```
|
||||
|
||||
## ConfigMap and Secrets
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: app-config
|
||||
data:
|
||||
LOG_LEVEL: "info"
|
||||
API_TIMEOUT: "30s"
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: app-secrets
|
||||
type: Opaque
|
||||
stringData:
|
||||
database-url: "postgres://user:pass@host:5432/db"
|
||||
```
|
||||
|
||||
## Horizontal Pod Autoscaler
|
||||
|
||||
```yaml
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: app-hpa
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: app
|
||||
minReplicas: 2
|
||||
maxReplicas: 10
|
||||
metrics:
|
||||
- type: Resource
|
||||
resource:
|
||||
name: cpu
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 70
|
||||
```
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Resource | Purpose |
|
||||
|----------|---------|
|
||||
| Deployment | Manages ReplicaSets, rolling updates |
|
||||
| Service | Internal load balancing, DNS |
|
||||
| Ingress | External HTTP/HTTPS routing |
|
||||
| ConfigMap | Non-sensitive configuration |
|
||||
| Secret | Sensitive data (base64 encoded) |
|
||||
| HPA | Auto-scaling based on metrics |
|
||||
| PVC | Persistent storage claims |
|
||||
|
||||
## Common kubectl Commands
|
||||
|
||||
```bash
|
||||
kubectl apply -f deployment.yaml
|
||||
kubectl get pods -l app=app
|
||||
kubectl describe pod <pod-name>
|
||||
kubectl logs -f <pod-name>
|
||||
kubectl exec -it <pod-name> -- /bin/sh
|
||||
kubectl rollout status deployment/app
|
||||
kubectl rollout undo deployment/app
|
||||
```
|
||||
@@ -0,0 +1,417 @@
|
||||
# Platform Engineering
|
||||
|
||||
## Platform Principles
|
||||
|
||||
- **Self-service first**: Reduce manual work to <10%
|
||||
- **Golden paths**: Pre-approved, opinionated templates
|
||||
- **Developer experience**: Measure and optimize productivity
|
||||
- **Platform as product**: Treat with product mindset
|
||||
|
||||
## Self-Service with Crossplane
|
||||
|
||||
```yaml
|
||||
# Composition for self-service database
|
||||
apiVersion: apiextensions.crossplane.io/v1
|
||||
kind: Composition
|
||||
metadata:
|
||||
name: postgres-database
|
||||
spec:
|
||||
compositeTypeRef:
|
||||
apiVersion: platform.example.com/v1alpha1
|
||||
kind: Database
|
||||
resources:
|
||||
- name: rds-instance
|
||||
base:
|
||||
apiVersion: rds.aws.crossplane.io/v1alpha1
|
||||
kind: DBInstance
|
||||
spec:
|
||||
forProvider:
|
||||
dbInstanceClass: db.t3.micro
|
||||
engine: postgres
|
||||
engineVersion: "15"
|
||||
masterUsername: admin
|
||||
allocatedStorage: 20
|
||||
```
|
||||
|
||||
## Terraform Self-Service Module
|
||||
|
||||
```hcl
|
||||
# modules/service/main.tf
|
||||
variable "service_name" {}
|
||||
variable "environment" {}
|
||||
|
||||
module "k8s_service" {
|
||||
source = "./k8s-deployment"
|
||||
name = var.service_name
|
||||
env = var.environment
|
||||
}
|
||||
|
||||
module "database" {
|
||||
source = "./postgres"
|
||||
name = "${var.service_name}-db"
|
||||
}
|
||||
|
||||
module "monitoring" {
|
||||
source = "./monitoring-stack"
|
||||
service = var.service_name
|
||||
}
|
||||
|
||||
output "service_url" {
|
||||
value = module.k8s_service.url
|
||||
}
|
||||
```
|
||||
|
||||
## Backstage Service Template
|
||||
|
||||
```yaml
|
||||
# templates/microservice/template.yaml
|
||||
apiVersion: scaffolder.backstage.io/v1beta3
|
||||
kind: Template
|
||||
metadata:
|
||||
name: microservice-template
|
||||
title: Microservice Golden Path
|
||||
spec:
|
||||
owner: platform-team
|
||||
type: service
|
||||
parameters:
|
||||
- title: Service Info
|
||||
properties:
|
||||
name:
|
||||
type: string
|
||||
owner:
|
||||
type: string
|
||||
ui:field: OwnerPicker
|
||||
language:
|
||||
type: string
|
||||
enum: [go, python, nodejs, java]
|
||||
steps:
|
||||
- id: fetch
|
||||
action: fetch:template
|
||||
input:
|
||||
url: ./skeleton
|
||||
values:
|
||||
name: ${{ parameters.name }}
|
||||
- id: publish
|
||||
action: publish:github
|
||||
input:
|
||||
repoUrl: github.com?owner=org&repo=${{ parameters.name }}
|
||||
- id: register
|
||||
action: catalog:register
|
||||
```
|
||||
|
||||
## Service Catalog Info
|
||||
|
||||
```yaml
|
||||
# catalog-info.yaml
|
||||
apiVersion: backstage.io/v1alpha1
|
||||
kind: Component
|
||||
metadata:
|
||||
name: payment-service
|
||||
annotations:
|
||||
github.com/project-slug: org/payment-service
|
||||
pagerduty.com/integration-key: abc123
|
||||
grafana/dashboard-selector: service=payment
|
||||
spec:
|
||||
type: service
|
||||
lifecycle: production
|
||||
owner: payments-team
|
||||
system: checkout
|
||||
dependsOn:
|
||||
- resource:default/payment-db
|
||||
- component:default/auth-service
|
||||
providesApis:
|
||||
- payment-api
|
||||
```
|
||||
|
||||
## Golden Path Scaffolding
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# create-service.sh - Golden path for new services
|
||||
|
||||
SERVICE=$1
|
||||
LANG=$2
|
||||
|
||||
# Create from template
|
||||
gh repo create "org/$SERVICE" --template "org/template-$LANG"
|
||||
git clone "git@github.com:org/$SERVICE.git"
|
||||
cd "$SERVICE"
|
||||
|
||||
# Setup CI/CD
|
||||
cat > .github/workflows/ci.yml <<EOF
|
||||
name: CI/CD
|
||||
on: [push]
|
||||
jobs:
|
||||
pipeline:
|
||||
uses: org/workflows/.github/workflows/standard.yml@v1
|
||||
with:
|
||||
service_name: $SERVICE
|
||||
EOF
|
||||
|
||||
# Create infrastructure
|
||||
cat > terraform/main.tf <<EOF
|
||||
module "service" {
|
||||
source = "git::https://github.com/org/terraform//service"
|
||||
name = "$SERVICE"
|
||||
}
|
||||
EOF
|
||||
|
||||
git add . && git commit -m "Golden path init" && git push
|
||||
|
||||
echo "✓ Service created! Merge to main to deploy."
|
||||
```
|
||||
|
||||
## GitOps Repository Structure
|
||||
|
||||
```
|
||||
gitops/
|
||||
├── apps/
|
||||
│ ├── production/
|
||||
│ │ ├── payment-service/
|
||||
│ │ └── auth-service/
|
||||
│ └── staging/
|
||||
│ └── payment-service/
|
||||
├── infrastructure/
|
||||
│ ├── clusters/
|
||||
│ │ ├── prod-us-east/
|
||||
│ │ └── prod-eu-west/
|
||||
│ └── base/
|
||||
│ ├── ingress/
|
||||
│ └── monitoring/
|
||||
└── platform/
|
||||
├── backstage/
|
||||
├── argocd/
|
||||
└── vault/
|
||||
```
|
||||
|
||||
## ArgoCD Application
|
||||
|
||||
```yaml
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: payment-service
|
||||
spec:
|
||||
project: default
|
||||
source:
|
||||
repoURL: https://github.com/org/gitops
|
||||
path: apps/production/payment-service
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
namespace: production
|
||||
syncPolicy:
|
||||
automated:
|
||||
prune: true
|
||||
selfHeal: true
|
||||
retry:
|
||||
limit: 5
|
||||
backoff:
|
||||
duration: 5s
|
||||
maxDuration: 3m
|
||||
```
|
||||
|
||||
## Platform Metrics
|
||||
|
||||
```yaml
|
||||
# prometheus/platform-metrics.yaml
|
||||
groups:
|
||||
- name: platform
|
||||
rules:
|
||||
# Self-service adoption rate
|
||||
- record: platform:self_service:rate
|
||||
expr: |
|
||||
sum(rate(platform_provision_automated[1h]))
|
||||
/
|
||||
sum(rate(platform_provision_total[1h]))
|
||||
|
||||
# Provisioning time P95
|
||||
- record: platform:provision:p95
|
||||
expr: |
|
||||
histogram_quantile(0.95,
|
||||
rate(platform_provision_duration_bucket[5m]))
|
||||
|
||||
# Golden path adoption
|
||||
- record: platform:golden_path:adoption
|
||||
expr: |
|
||||
count(service{template="golden-path"})
|
||||
/ count(service)
|
||||
```
|
||||
|
||||
## Custom Backstage Plugin
|
||||
|
||||
```typescript
|
||||
// plugins/platform-stats/PlatformMetrics.tsx
|
||||
import React from 'react';
|
||||
import { InfoCard, Progress } from '@backstage/core-components';
|
||||
|
||||
export const PlatformMetrics = () => {
|
||||
const metrics = {
|
||||
selfServiceRate: 92,
|
||||
avgProvisionTime: '3.5min',
|
||||
uptime: '99.95%',
|
||||
satisfaction: 4.6
|
||||
};
|
||||
|
||||
return (
|
||||
<InfoCard title="Platform Health">
|
||||
<Progress value={metrics.selfServiceRate} label="Self-Service" />
|
||||
<p>Provision Time: {metrics.avgProvisionTime}</p>
|
||||
<p>Uptime: {metrics.uptime}</p>
|
||||
<p>Satisfaction: {metrics.satisfaction}/5</p>
|
||||
</InfoCard>
|
||||
);
|
||||
};
|
||||
```
|
||||
|
||||
## Cost Allocation
|
||||
|
||||
```yaml
|
||||
# kubecost/allocation.yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: cost-allocation
|
||||
data:
|
||||
allocation.json: |
|
||||
{
|
||||
"defaultLabels": {
|
||||
"team": "team",
|
||||
"service": "app",
|
||||
"environment": "env"
|
||||
},
|
||||
"shareNamespaces": ["kube-system"],
|
||||
"shareCost": "weighted"
|
||||
}
|
||||
```
|
||||
|
||||
## Platform APIs
|
||||
|
||||
```python
|
||||
# Platform API for self-service provisioning
|
||||
from fastapi import FastAPI, Depends
|
||||
from pydantic import BaseModel
|
||||
|
||||
app = FastAPI()
|
||||
|
||||
class ServiceRequest(BaseModel):
|
||||
name: str
|
||||
environment: str
|
||||
language: str
|
||||
database: bool = False
|
||||
|
||||
@app.post("/api/v1/services")
|
||||
async def create_service(request: ServiceRequest):
|
||||
# Validate and enqueue
|
||||
task = platform.provision_service(
|
||||
name=request.name,
|
||||
env=request.environment,
|
||||
template=f"golden-path-{request.language}"
|
||||
)
|
||||
return {"task_id": task.id, "status": "provisioning"}
|
||||
|
||||
@app.get("/api/v1/services/{name}/status")
|
||||
async def service_status(name: str):
|
||||
return {
|
||||
"status": "running",
|
||||
"url": f"https://{name}.example.com",
|
||||
"health": "healthy",
|
||||
"cost_mtd": "$142.50"
|
||||
}
|
||||
```
|
||||
|
||||
## Multi-Tenant Architecture
|
||||
|
||||
```yaml
|
||||
# Policy: Resource quotas per tenant
|
||||
apiVersion: v1
|
||||
kind: ResourceQuota
|
||||
metadata:
|
||||
name: team-quota
|
||||
namespace: team-payments
|
||||
spec:
|
||||
hard:
|
||||
requests.cpu: "20"
|
||||
requests.memory: 40Gi
|
||||
persistentvolumeclaims: "10"
|
||||
services.loadbalancers: "2"
|
||||
---
|
||||
# RBAC: Namespace admin
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: team-admin
|
||||
namespace: team-payments
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: namespace-admin
|
||||
subjects:
|
||||
- kind: Group
|
||||
name: team-payments
|
||||
```
|
||||
|
||||
## Adoption Strategy
|
||||
|
||||
```yaml
|
||||
# Platform metrics tracking
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: platform-goals
|
||||
data:
|
||||
goals.yaml: |
|
||||
q1_2024:
|
||||
self_service_rate: 90%
|
||||
avg_provision_time: 5min
|
||||
developer_satisfaction: 4.5/5
|
||||
golden_path_adoption: 80%
|
||||
|
||||
tracking:
|
||||
weekly_provisioning: true
|
||||
team_feedback: true
|
||||
support_tickets: true
|
||||
training_completion: true
|
||||
```
|
||||
|
||||
## CLI Tool Example
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# platform-cli - Self-service CLI
|
||||
|
||||
platform() {
|
||||
case $1 in
|
||||
create)
|
||||
curl -X POST $PLATFORM_API/services \
|
||||
-d "{\"name\":\"$2\",\"env\":\"$3\",\"language\":\"$4\"}"
|
||||
;;
|
||||
status)
|
||||
curl $PLATFORM_API/services/$2/status | jq
|
||||
;;
|
||||
logs)
|
||||
kubectl logs -l app=$2 -n ${3:-staging} --tail=100
|
||||
;;
|
||||
cost)
|
||||
curl $PLATFORM_API/services/$2/cost?period=mtd
|
||||
;;
|
||||
esac
|
||||
}
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Design for self-service from day one
|
||||
- Make golden paths the easiest option
|
||||
- Measure developer satisfaction continuously
|
||||
- Automate platform operations
|
||||
- Provide excellent documentation
|
||||
- Build APIs, not just tools
|
||||
- Enable safe experimentation
|
||||
- Maintain backward compatibility
|
||||
- Treat platform as a product
|
||||
- Gather and act on feedback
|
||||
- Track adoption metrics weekly
|
||||
- Run platform as a product team
|
||||
- Invest in developer evangelism
|
||||
- Maintain SLOs for platform uptime
|
||||
- Provide fast, helpful support
|
||||
@@ -0,0 +1,527 @@
|
||||
# Release Automation
|
||||
|
||||
## Artifact Management
|
||||
|
||||
### Container Registry Lifecycle
|
||||
|
||||
```json
|
||||
{
|
||||
"rules": [
|
||||
{
|
||||
"rulePriority": 1,
|
||||
"description": "Keep last 10 prod images",
|
||||
"selection": {
|
||||
"tagStatus": "tagged",
|
||||
"tagPrefixList": ["prod-"],
|
||||
"countType": "imageCountMoreThan",
|
||||
"countNumber": 10
|
||||
},
|
||||
"action": {"type": "expire"}
|
||||
},
|
||||
{
|
||||
"rulePriority": 2,
|
||||
"description": "Remove untagged after 7 days",
|
||||
"selection": {
|
||||
"tagStatus": "untagged",
|
||||
"countType": "sinceImagePushed",
|
||||
"countUnit": "days",
|
||||
"countNumber": 7
|
||||
},
|
||||
"action": {"type": "expire"}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Artifact Promotion
|
||||
|
||||
```yaml
|
||||
# .github/workflows/promote.yml
|
||||
name: Artifact Promotion
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
image_tag:
|
||||
required: true
|
||||
target_env:
|
||||
type: choice
|
||||
options: [staging, production]
|
||||
|
||||
jobs:
|
||||
promote:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Re-tag for environment
|
||||
run: |
|
||||
docker pull $REGISTRY/$IMAGE:${{ inputs.image_tag }}
|
||||
docker tag $REGISTRY/$IMAGE:${{ inputs.image_tag }} \
|
||||
$REGISTRY/$IMAGE:${{ inputs.target_env }}-latest
|
||||
docker push $REGISTRY/$IMAGE:${{ inputs.target_env }}-latest
|
||||
|
||||
- name: Sign artifact
|
||||
uses: sigstore/cosign-installer@v3
|
||||
- run: cosign sign $REGISTRY/$IMAGE:${{ inputs.target_env }}-latest
|
||||
|
||||
- name: Update GitOps
|
||||
run: |
|
||||
cd gitops/apps/${{ inputs.target_env }}
|
||||
yq e '.image.tag = "${{ inputs.image_tag }}"' -i values.yaml
|
||||
git commit -am "Promote to ${{ inputs.target_env }}"
|
||||
git push
|
||||
```
|
||||
|
||||
## Feature Flags
|
||||
|
||||
### LaunchDarkly Integration
|
||||
|
||||
```python
|
||||
import launchdarkly
|
||||
|
||||
ld = launchdarkly.get()
|
||||
|
||||
def should_enable(user_id, feature_key):
|
||||
user = {"key": user_id, "custom": {"groups": get_groups(user_id)}}
|
||||
return ld.variation(feature_key, user, False)
|
||||
|
||||
# Usage
|
||||
if should_enable(user.id, "new-payment-flow"):
|
||||
return new_payment_service.process(payment)
|
||||
else:
|
||||
return legacy_payment_service.process(payment)
|
||||
```
|
||||
|
||||
### Flagger Progressive Delivery
|
||||
|
||||
```yaml
|
||||
apiVersion: flagger.app/v1beta1
|
||||
kind: Canary
|
||||
metadata:
|
||||
name: payment-service
|
||||
spec:
|
||||
targetRef:
|
||||
kind: Deployment
|
||||
name: payment-service
|
||||
service:
|
||||
port: 8080
|
||||
analysis:
|
||||
interval: 1m
|
||||
threshold: 5
|
||||
maxWeight: 50
|
||||
stepWeight: 10
|
||||
metrics:
|
||||
- name: request-success-rate
|
||||
thresholdRange:
|
||||
min: 99
|
||||
- name: request-duration
|
||||
thresholdRange:
|
||||
max: 500
|
||||
webhooks:
|
||||
- name: load-test
|
||||
url: http://flagger-loadtester/
|
||||
metadata:
|
||||
cmd: "hey -z 1m -q 10 http://payment-canary/"
|
||||
```
|
||||
|
||||
## Multi-Platform CI/CD
|
||||
|
||||
### GitLab CI
|
||||
|
||||
```yaml
|
||||
stages: [test, build, deploy]
|
||||
|
||||
test:
|
||||
stage: test
|
||||
image: node:20
|
||||
script:
|
||||
- npm ci && npm test
|
||||
|
||||
build:
|
||||
stage: build
|
||||
image: docker:latest
|
||||
services: [docker:dind]
|
||||
script:
|
||||
- docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
|
||||
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
|
||||
|
||||
deploy:production:
|
||||
stage: deploy
|
||||
script:
|
||||
- kubectl set image deployment/app app=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
|
||||
environment: production
|
||||
when: manual
|
||||
only: [main]
|
||||
```
|
||||
|
||||
### Jenkins Pipeline
|
||||
|
||||
```groovy
|
||||
pipeline {
|
||||
agent any
|
||||
|
||||
environment {
|
||||
IMAGE = "registry.example.com/app"
|
||||
}
|
||||
|
||||
stages {
|
||||
stage('Test') {
|
||||
steps {
|
||||
sh 'npm ci && npm test'
|
||||
junit 'reports/junit.xml'
|
||||
}
|
||||
}
|
||||
|
||||
stage('Build') {
|
||||
steps {
|
||||
script {
|
||||
docker.build("${IMAGE}:${BUILD_NUMBER}")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
stage('Security Scan') {
|
||||
steps {
|
||||
sh "trivy image ${IMAGE}:${BUILD_NUMBER}"
|
||||
}
|
||||
}
|
||||
|
||||
stage('Deploy Staging') {
|
||||
when { branch 'main' }
|
||||
steps {
|
||||
sh "kubectl set image deployment/app app=${IMAGE}:${BUILD_NUMBER} -n staging"
|
||||
}
|
||||
}
|
||||
|
||||
stage('Deploy Production') {
|
||||
when { branch 'main' }
|
||||
steps {
|
||||
input 'Deploy to production?'
|
||||
sh "kubectl set image deployment/app app=${IMAGE}:${BUILD_NUMBER} -n production"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
post {
|
||||
failure {
|
||||
slackSend color: 'danger', message: "Build failed: ${JOB_NAME}"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Build Optimization
|
||||
|
||||
### Multi-stage Docker Build
|
||||
|
||||
```dockerfile
|
||||
FROM node:20 AS deps
|
||||
WORKDIR /app
|
||||
COPY package*.json ./
|
||||
RUN npm ci --only=production
|
||||
|
||||
FROM node:20 AS builder
|
||||
WORKDIR /app
|
||||
COPY package*.json ./
|
||||
RUN npm ci
|
||||
COPY . .
|
||||
RUN npm run build
|
||||
|
||||
FROM node:20-slim AS runner
|
||||
WORKDIR /app
|
||||
ENV NODE_ENV production
|
||||
COPY --from=deps /app/node_modules ./node_modules
|
||||
COPY --from=builder /app/dist ./dist
|
||||
USER node
|
||||
CMD ["node", "dist/main.js"]
|
||||
```
|
||||
|
||||
### Parallel Testing
|
||||
|
||||
```yaml
|
||||
# CircleCI
|
||||
version: 2.1
|
||||
jobs:
|
||||
test:
|
||||
parallelism: 4
|
||||
docker:
|
||||
- image: cimg/node:20
|
||||
steps:
|
||||
- checkout
|
||||
- run: npm ci
|
||||
- run: |
|
||||
TESTS=$(circleci tests glob "test/**/*.js" | circleci tests split)
|
||||
npm test $TESTS
|
||||
```
|
||||
|
||||
## Release Orchestration
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# release.sh - Multi-service coordinated release
|
||||
|
||||
VERSION=$1
|
||||
SERVICES=(auth api worker frontend)
|
||||
|
||||
echo "Release: $VERSION"
|
||||
|
||||
# Create release branches
|
||||
for svc in "${SERVICES[@]}"; do
|
||||
gh api repos/org/$svc/git/refs -f ref=refs/heads/release/$VERSION -f sha=$(git rev-parse main)
|
||||
done
|
||||
|
||||
# Trigger builds
|
||||
for svc in "${SERVICES[@]}"; do
|
||||
gh workflow run ci.yml --repo org/$svc --ref release/$VERSION
|
||||
done
|
||||
|
||||
# Wait for completion
|
||||
for svc in "${SERVICES[@]}"; do
|
||||
gh run watch --repo org/$svc $(gh run list --repo org/$svc -L1 -q '.[0].databaseId')
|
||||
done
|
||||
|
||||
# Deploy to staging
|
||||
kubectl apply -f staging/release-$VERSION.yaml
|
||||
|
||||
# Smoke tests
|
||||
./scripts/smoke-test.sh staging
|
||||
|
||||
echo "✓ Release $VERSION ready for production"
|
||||
```
|
||||
|
||||
## Dependency Management
|
||||
|
||||
### Renovate Auto-Update
|
||||
|
||||
```json
|
||||
{
|
||||
"extends": ["config:base"],
|
||||
"packageRules": [
|
||||
{
|
||||
"matchUpdateTypes": ["minor", "patch"],
|
||||
"automerge": true
|
||||
},
|
||||
{
|
||||
"matchDepTypes": ["devDependencies"],
|
||||
"automerge": true
|
||||
}
|
||||
],
|
||||
"schedule": ["before 6am on Monday"],
|
||||
"prConcurrentLimit": 5
|
||||
}
|
||||
```
|
||||
|
||||
## Build Optimization
|
||||
|
||||
### Build Caching Strategy
|
||||
|
||||
```yaml
|
||||
# GitHub Actions: Multi-layer caching
|
||||
- name: Cache dependencies
|
||||
uses: actions/cache@v3
|
||||
with:
|
||||
path: |
|
||||
~/.npm
|
||||
~/.cache
|
||||
node_modules
|
||||
key: ${{ runner.os }}-deps-${{ hashFiles('**/package-lock.json') }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-deps-
|
||||
|
||||
- name: Cache Docker layers
|
||||
uses: docker/build-push-action@v4
|
||||
with:
|
||||
context: .
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
```
|
||||
|
||||
### Parallel CI Pipeline
|
||||
|
||||
```yaml
|
||||
# Multi-platform builds in parallel
|
||||
name: Build
|
||||
|
||||
on: [push]
|
||||
|
||||
jobs:
|
||||
test:
|
||||
strategy:
|
||||
matrix:
|
||||
node: [18, 20, 22]
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- uses: actions/setup-node@v3
|
||||
with:
|
||||
node-version: ${{ matrix.node }}
|
||||
- run: npm ci && npm test
|
||||
|
||||
build-images:
|
||||
needs: test
|
||||
strategy:
|
||||
matrix:
|
||||
platform: [linux/amd64, linux/arm64]
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: docker/build-push-action@v4
|
||||
with:
|
||||
platforms: ${{ matrix.platform }}
|
||||
tags: app:${{ github.sha }}
|
||||
```
|
||||
|
||||
## Multi-Service Release Orchestration
|
||||
|
||||
```yaml
|
||||
# release-coordinator.yaml
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: release-v2.5.0
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: coordinator
|
||||
image: release-bot:latest
|
||||
env:
|
||||
- name: RELEASE_VERSION
|
||||
value: "v2.5.0"
|
||||
- name: SERVICES
|
||||
value: "auth,api,worker,frontend"
|
||||
command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
# Deploy in dependency order
|
||||
for svc in auth api worker frontend; do
|
||||
echo "Deploying $svc..."
|
||||
kubectl set image deploy/$svc \
|
||||
$svc=registry.io/$svc:$RELEASE_VERSION
|
||||
|
||||
kubectl rollout status deploy/$svc --timeout=5m
|
||||
|
||||
# Health check
|
||||
kubectl run test-$svc --rm -i --restart=Never \
|
||||
--image=curlimages/curl -- \
|
||||
curl -f http://$svc/health
|
||||
|
||||
echo "$svc deployed successfully"
|
||||
done
|
||||
```
|
||||
|
||||
## Advanced Artifact Management
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# artifact-scanner.sh - Scan before promotion
|
||||
|
||||
IMAGE=$1
|
||||
SEVERITY=${2:-HIGH}
|
||||
|
||||
# Vulnerability scan
|
||||
trivy image --severity $SEVERITY --exit-code 1 $IMAGE
|
||||
|
||||
# License compliance
|
||||
syft $IMAGE -o json | \
|
||||
jq '.artifacts[].licenses[] | select(.value |
|
||||
contains("GPL") or contains("AGPL"))' && \
|
||||
echo "License violation detected" && exit 1
|
||||
|
||||
# SBOM generation
|
||||
syft $IMAGE -o spdx-json > sbom-$(basename $IMAGE).spdx.json
|
||||
|
||||
# Sign artifact
|
||||
cosign sign --key cosign.key $IMAGE
|
||||
|
||||
# Promote
|
||||
docker tag $IMAGE $IMAGE-approved
|
||||
docker push $IMAGE-approved
|
||||
|
||||
echo "Artifact $IMAGE approved and promoted"
|
||||
```
|
||||
|
||||
## Zero-Downtime Database Migrations
|
||||
|
||||
```python
|
||||
# migrations/release_v2.5.py
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
def upgrade():
|
||||
# Step 1: Add new column (nullable)
|
||||
op.add_column('users',
|
||||
sa.Column('email_verified', sa.Boolean(), nullable=True))
|
||||
|
||||
# Step 2: Backfill data (in batches)
|
||||
connection = op.get_bind()
|
||||
connection.execute("""
|
||||
UPDATE users SET email_verified = true
|
||||
WHERE email IS NOT NULL
|
||||
LIMIT 1000
|
||||
""")
|
||||
# Repeat until complete (or use background job)
|
||||
|
||||
# Step 3: Make non-nullable (in next release)
|
||||
# op.alter_column('users', 'email_verified', nullable=False)
|
||||
|
||||
def downgrade():
|
||||
op.drop_column('users', 'email_verified')
|
||||
```
|
||||
|
||||
## Release Metrics Dashboard
|
||||
|
||||
```yaml
|
||||
# Grafana dashboard for release metrics
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: release-dashboard
|
||||
data:
|
||||
dashboard.json: |
|
||||
{
|
||||
"panels": [
|
||||
{
|
||||
"title": "Deployment Frequency",
|
||||
"targets": [{
|
||||
"expr": "count_over_time(deployment_completed[1d])"
|
||||
}]
|
||||
},
|
||||
{
|
||||
"title": "Lead Time",
|
||||
"targets": [{
|
||||
"expr": "histogram_quantile(0.95, commit_to_deploy_seconds_bucket)"
|
||||
}]
|
||||
},
|
||||
{
|
||||
"title": "Change Failure Rate",
|
||||
"targets": [{
|
||||
"expr": "sum(rate(deployment_failed[1h])) / sum(rate(deployment_total[1h]))"
|
||||
}]
|
||||
},
|
||||
{
|
||||
"title": "Active Releases",
|
||||
"targets": [{
|
||||
"expr": "count(release_in_progress == 1)"
|
||||
}]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Version artifacts with immutable tags
|
||||
- Implement retention policies
|
||||
- Use progressive delivery for high-risk changes
|
||||
- Automate security scanning
|
||||
- Maintain deployment audit trails
|
||||
- Enable easy rollbacks
|
||||
- Monitor deployment metrics
|
||||
- Use feature flags for flexibility
|
||||
- Cache aggressively for fast builds
|
||||
- Parallelize test and build jobs
|
||||
- Coordinate multi-service releases
|
||||
- Generate and track SBOMs
|
||||
- Sign artifacts for supply chain security
|
||||
- Automate dependency updates
|
||||
- Track DORA metrics continuously
|
||||
@@ -0,0 +1,141 @@
|
||||
# Terraform Infrastructure as Code
|
||||
|
||||
## AWS ECS Fargate Setup
|
||||
|
||||
```hcl
|
||||
terraform {
|
||||
required_providers {
|
||||
aws = { source = "hashicorp/aws", version = "~> 5.0" }
|
||||
}
|
||||
backend "s3" {
|
||||
bucket = "terraform-state"
|
||||
key = "app/terraform.tfstate"
|
||||
region = "us-east-1"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_ecs_cluster" "main" {
|
||||
name = "app-cluster"
|
||||
setting {
|
||||
name = "containerInsights"
|
||||
value = "enabled"
|
||||
}
|
||||
}
|
||||
|
||||
resource "aws_ecs_task_definition" "app" {
|
||||
family = "app"
|
||||
network_mode = "awsvpc"
|
||||
requires_compatibilities = ["FARGATE"]
|
||||
cpu = "256"
|
||||
memory = "512"
|
||||
execution_role_arn = aws_iam_role.ecs_execution.arn
|
||||
|
||||
container_definitions = jsonencode([{
|
||||
name = "app"
|
||||
image = "${var.ecr_repository}:${var.image_tag}"
|
||||
portMappings = [{ containerPort = 3000 }]
|
||||
logConfiguration = {
|
||||
logDriver = "awslogs"
|
||||
options = {
|
||||
awslogs-group = aws_cloudwatch_log_group.app.name
|
||||
awslogs-region = var.region
|
||||
awslogs-stream-prefix = "app"
|
||||
}
|
||||
}
|
||||
secrets = [
|
||||
{ name = "DATABASE_URL", valueFrom = aws_ssm_parameter.db_url.arn }
|
||||
]
|
||||
}])
|
||||
}
|
||||
|
||||
resource "aws_ecs_service" "app" {
|
||||
name = "app"
|
||||
cluster = aws_ecs_cluster.main.id
|
||||
task_definition = aws_ecs_task_definition.app.arn
|
||||
desired_count = 2
|
||||
launch_type = "FARGATE"
|
||||
|
||||
network_configuration {
|
||||
subnets = var.private_subnets
|
||||
security_groups = [aws_security_group.app.id]
|
||||
}
|
||||
|
||||
load_balancer {
|
||||
target_group_arn = aws_lb_target_group.app.arn
|
||||
container_name = "app"
|
||||
container_port = 3000
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## AWS RDS PostgreSQL
|
||||
|
||||
```hcl
|
||||
resource "aws_db_instance" "postgres" {
|
||||
identifier = "app-db"
|
||||
engine = "postgres"
|
||||
engine_version = "16.1"
|
||||
instance_class = "db.t3.micro"
|
||||
allocated_storage = 20
|
||||
storage_encrypted = true
|
||||
|
||||
db_name = "app"
|
||||
username = "admin"
|
||||
password = var.db_password
|
||||
|
||||
vpc_security_group_ids = [aws_security_group.db.id]
|
||||
db_subnet_group_name = aws_db_subnet_group.main.name
|
||||
|
||||
backup_retention_period = 7
|
||||
skip_final_snapshot = false
|
||||
final_snapshot_identifier = "app-db-final"
|
||||
|
||||
tags = { Environment = var.environment }
|
||||
}
|
||||
```
|
||||
|
||||
## Variables and Outputs
|
||||
|
||||
```hcl
|
||||
# variables.tf
|
||||
variable "environment" {
|
||||
type = string
|
||||
description = "Environment name"
|
||||
}
|
||||
|
||||
variable "region" {
|
||||
type = string
|
||||
default = "us-east-1"
|
||||
}
|
||||
|
||||
# outputs.tf
|
||||
output "ecs_cluster_arn" {
|
||||
value = aws_ecs_cluster.main.arn
|
||||
}
|
||||
|
||||
output "alb_dns_name" {
|
||||
value = aws_lb.main.dns_name
|
||||
}
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
| Practice | Implementation |
|
||||
|----------|----------------|
|
||||
| State locking | S3 backend with DynamoDB |
|
||||
| Secrets | Use AWS Secrets Manager / SSM |
|
||||
| Modules | Reusable components |
|
||||
| Workspaces | Environment separation |
|
||||
| Tagging | Consistent resource tags |
|
||||
| Validation | `terraform validate`, `tflint` |
|
||||
|
||||
## Common Commands
|
||||
|
||||
```bash
|
||||
terraform init
|
||||
terraform plan -out=tfplan
|
||||
terraform apply tfplan
|
||||
terraform destroy
|
||||
terraform state list
|
||||
terraform import aws_instance.app i-1234567890
|
||||
```
|
||||
@@ -0,0 +1,404 @@
|
||||
---
|
||||
name: reverse-proxy
|
||||
description: Configure nginx and Traefik as reverse proxies. Implement SSL termination and routing. Use when setting up application gateways.
|
||||
license: MIT
|
||||
metadata:
|
||||
author: devops-skills
|
||||
version: "1.0"
|
||||
---
|
||||
|
||||
# Reverse Proxy
|
||||
|
||||
Configure reverse proxies to route traffic, terminate TLS, enforce rate limits, and serve as the gateway between clients and backend services.
|
||||
|
||||
## When to Use
|
||||
|
||||
- Routing traffic from a public domain to one or more backend services.
|
||||
- Terminating TLS at the edge and forwarding plain HTTP to backends.
|
||||
- Adding rate limiting, CORS, security headers, and access control.
|
||||
- Consolidating multiple services under a single domain with path-based routing.
|
||||
- Handling WebSocket upgrades, gRPC proxying, or HTTP/2 passthrough.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Backend service(s) running on known host:port.
|
||||
- TLS certificate (Let's Encrypt, ACM, or self-signed for development).
|
||||
- nginx 1.25+ or Traefik 3.x installed.
|
||||
- DNS record pointing the domain to the proxy server.
|
||||
|
||||
## nginx Reverse Proxy
|
||||
|
||||
### Basic HTTPS Proxy with Redirect
|
||||
|
||||
```nginx
|
||||
# /etc/nginx/sites-available/app.example.com
|
||||
server {
|
||||
listen 80;
|
||||
server_name app.example.com;
|
||||
return 301 https://$host$request_uri;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl http2;
|
||||
server_name app.example.com;
|
||||
|
||||
# TLS configuration
|
||||
ssl_certificate /etc/letsencrypt/live/app.example.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/app.example.com/privkey.pem;
|
||||
ssl_protocols TLSv1.2 TLSv1.3;
|
||||
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384;
|
||||
ssl_prefer_server_ciphers on;
|
||||
ssl_session_cache shared:SSL:10m;
|
||||
ssl_session_timeout 10m;
|
||||
|
||||
# Security headers
|
||||
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;
|
||||
add_header X-Frame-Options DENY always;
|
||||
add_header X-Content-Type-Options nosniff always;
|
||||
add_header Referrer-Policy strict-origin-when-cross-origin always;
|
||||
|
||||
# Proxy to backend
|
||||
location / {
|
||||
proxy_pass http://127.0.0.1:3000;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
# Timeouts
|
||||
proxy_connect_timeout 5s;
|
||||
proxy_read_timeout 60s;
|
||||
proxy_send_timeout 60s;
|
||||
|
||||
# Buffering
|
||||
proxy_buffering on;
|
||||
proxy_buffer_size 4k;
|
||||
proxy_buffers 8 4k;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Path-Based Routing to Multiple Services
|
||||
|
||||
```nginx
|
||||
server {
|
||||
listen 443 ssl http2;
|
||||
server_name app.example.com;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/app.example.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/app.example.com/privkey.pem;
|
||||
|
||||
# Frontend SPA
|
||||
location / {
|
||||
proxy_pass http://127.0.0.1:3000;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
# API backend
|
||||
location /api/ {
|
||||
proxy_pass http://127.0.0.1:8080/;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_read_timeout 120s;
|
||||
}
|
||||
|
||||
# WebSocket endpoint
|
||||
location /ws/ {
|
||||
proxy_pass http://127.0.0.1:8080;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_set_header Host $host;
|
||||
proxy_read_timeout 86400s; # 24h for long-lived connections
|
||||
}
|
||||
|
||||
# Static assets with caching
|
||||
location /static/ {
|
||||
alias /var/www/static/;
|
||||
expires 30d;
|
||||
add_header Cache-Control "public, immutable";
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Rate Limiting
|
||||
|
||||
```nginx
|
||||
# Define rate limit zones in http block
|
||||
http {
|
||||
# 10 requests/second per IP
|
||||
limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s;
|
||||
|
||||
# 1 request/second for login
|
||||
limit_req_zone $binary_remote_addr zone=login_limit:10m rate=1r/s;
|
||||
|
||||
# Connection limit per IP
|
||||
limit_conn_zone $binary_remote_addr zone=conn_limit:10m;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl http2;
|
||||
server_name app.example.com;
|
||||
|
||||
# Apply rate limit to API
|
||||
location /api/ {
|
||||
limit_req zone=api_limit burst=20 nodelay;
|
||||
limit_req_status 429;
|
||||
proxy_pass http://127.0.0.1:8080;
|
||||
}
|
||||
|
||||
# Strict rate limit on auth endpoints
|
||||
location /api/auth/ {
|
||||
limit_req zone=login_limit burst=5;
|
||||
limit_req_status 429;
|
||||
proxy_pass http://127.0.0.1:8080;
|
||||
}
|
||||
|
||||
# Connection limit
|
||||
location / {
|
||||
limit_conn conn_limit 100;
|
||||
proxy_pass http://127.0.0.1:3000;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Gzip and Brotli Compression
|
||||
|
||||
```nginx
|
||||
http {
|
||||
gzip on;
|
||||
gzip_types text/plain text/css application/json application/javascript text/xml application/xml image/svg+xml;
|
||||
gzip_min_length 256;
|
||||
gzip_vary on;
|
||||
gzip_proxied any;
|
||||
gzip_comp_level 5;
|
||||
|
||||
# Brotli (requires ngx_brotli module)
|
||||
# brotli on;
|
||||
# brotli_types text/plain text/css application/json application/javascript text/xml application/xml image/svg+xml;
|
||||
# brotli_comp_level 6;
|
||||
}
|
||||
```
|
||||
|
||||
### Let's Encrypt with Certbot
|
||||
|
||||
```bash
|
||||
# Install certbot with nginx plugin
|
||||
sudo apt install certbot python3-certbot-nginx
|
||||
|
||||
# Obtain and install certificate
|
||||
sudo certbot --nginx -d app.example.com -d www.example.com
|
||||
|
||||
# Auto-renewal is configured via systemd timer
|
||||
sudo systemctl status certbot.timer
|
||||
|
||||
# Manual renewal test
|
||||
sudo certbot renew --dry-run
|
||||
```
|
||||
|
||||
## Traefik Reverse Proxy
|
||||
|
||||
### Static Configuration
|
||||
|
||||
```yaml
|
||||
# traefik.yml
|
||||
entryPoints:
|
||||
web:
|
||||
address: ":80"
|
||||
http:
|
||||
redirections:
|
||||
entryPoint:
|
||||
to: websecure
|
||||
scheme: https
|
||||
websecure:
|
||||
address: ":443"
|
||||
|
||||
certificatesResolvers:
|
||||
letsencrypt:
|
||||
acme:
|
||||
email: admin@example.com
|
||||
storage: /letsencrypt/acme.json
|
||||
httpChallenge:
|
||||
entryPoint: web
|
||||
|
||||
providers:
|
||||
docker:
|
||||
exposedByDefault: false
|
||||
file:
|
||||
directory: /etc/traefik/dynamic/
|
||||
|
||||
api:
|
||||
dashboard: true
|
||||
insecure: false
|
||||
|
||||
log:
|
||||
level: INFO
|
||||
|
||||
accessLog:
|
||||
filePath: /var/log/traefik/access.log
|
||||
```
|
||||
|
||||
### Dynamic Configuration (File Provider)
|
||||
|
||||
```yaml
|
||||
# /etc/traefik/dynamic/services.yml
|
||||
http:
|
||||
routers:
|
||||
app:
|
||||
rule: "Host(`app.example.com`)"
|
||||
entryPoints:
|
||||
- websecure
|
||||
service: app
|
||||
tls:
|
||||
certResolver: letsencrypt
|
||||
middlewares:
|
||||
- security-headers
|
||||
- rate-limit
|
||||
|
||||
api:
|
||||
rule: "Host(`app.example.com`) && PathPrefix(`/api`)"
|
||||
entryPoints:
|
||||
- websecure
|
||||
service: api
|
||||
tls:
|
||||
certResolver: letsencrypt
|
||||
|
||||
services:
|
||||
app:
|
||||
loadBalancer:
|
||||
servers:
|
||||
- url: "http://127.0.0.1:3000"
|
||||
healthCheck:
|
||||
path: /health
|
||||
interval: 10s
|
||||
timeout: 3s
|
||||
|
||||
api:
|
||||
loadBalancer:
|
||||
servers:
|
||||
- url: "http://127.0.0.1:8080"
|
||||
healthCheck:
|
||||
path: /api/health
|
||||
interval: 10s
|
||||
timeout: 3s
|
||||
|
||||
middlewares:
|
||||
security-headers:
|
||||
headers:
|
||||
stsSeconds: 63072000
|
||||
stsIncludeSubdomains: true
|
||||
frameDeny: true
|
||||
contentTypeNosniff: true
|
||||
browserXssFilter: true
|
||||
referrerPolicy: strict-origin-when-cross-origin
|
||||
|
||||
rate-limit:
|
||||
rateLimit:
|
||||
average: 100
|
||||
burst: 50
|
||||
period: 1m
|
||||
```
|
||||
|
||||
### Traefik with Docker Labels
|
||||
|
||||
```yaml
|
||||
# docker-compose.yml
|
||||
version: "3.8"
|
||||
|
||||
services:
|
||||
traefik:
|
||||
image: traefik:v3.0
|
||||
ports:
|
||||
- "80:80"
|
||||
- "443:443"
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
- ./traefik.yml:/etc/traefik/traefik.yml:ro
|
||||
- letsencrypt:/letsencrypt
|
||||
|
||||
frontend:
|
||||
image: my-frontend:latest
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.routers.frontend.rule=Host(`app.example.com`)"
|
||||
- "traefik.http.routers.frontend.tls.certresolver=letsencrypt"
|
||||
- "traefik.http.services.frontend.loadbalancer.server.port=3000"
|
||||
|
||||
api:
|
||||
image: my-api:latest
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.routers.api.rule=Host(`app.example.com`) && PathPrefix(`/api`)"
|
||||
- "traefik.http.routers.api.tls.certresolver=letsencrypt"
|
||||
- "traefik.http.services.api.loadbalancer.server.port=8080"
|
||||
- "traefik.http.routers.api.middlewares=api-ratelimit"
|
||||
- "traefik.http.middlewares.api-ratelimit.ratelimit.average=50"
|
||||
- "traefik.http.middlewares.api-ratelimit.ratelimit.burst=25"
|
||||
|
||||
volumes:
|
||||
letsencrypt:
|
||||
```
|
||||
|
||||
## nginx Testing and Management
|
||||
|
||||
```bash
|
||||
# Test configuration syntax
|
||||
sudo nginx -t
|
||||
|
||||
# Reload without downtime
|
||||
sudo nginx -s reload
|
||||
|
||||
# View active connections
|
||||
sudo nginx -s status
|
||||
|
||||
# Check which config file is active
|
||||
nginx -V 2>&1 | grep -o '\-\-conf-path=[^ ]*'
|
||||
|
||||
# Monitor access logs
|
||||
tail -f /var/log/nginx/access.log
|
||||
|
||||
# Monitor error logs
|
||||
tail -f /var/log/nginx/error.log
|
||||
```
|
||||
|
||||
## IP Allowlisting and Geoblocking
|
||||
|
||||
```nginx
|
||||
# Allow only specific IPs (admin panel)
|
||||
location /admin/ {
|
||||
allow 203.0.113.0/24;
|
||||
allow 198.51.100.5;
|
||||
deny all;
|
||||
proxy_pass http://127.0.0.1:3000;
|
||||
}
|
||||
|
||||
# Block by country (requires GeoIP2 module)
|
||||
# geoip2 /usr/share/GeoIP/GeoLite2-Country.mmdb {
|
||||
# auto_reload 60m;
|
||||
# $geoip2_data_country_iso_code country iso_code;
|
||||
# }
|
||||
# if ($geoip2_data_country_iso_code = "XX") {
|
||||
# return 403;
|
||||
# }
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---------|-------|-----|
|
||||
| 502 Bad Gateway | Backend not running or unreachable | Verify backend is listening; check `proxy_pass` URL |
|
||||
| 504 Gateway Timeout | Backend too slow | Increase `proxy_read_timeout`; check backend performance |
|
||||
| Mixed content warnings | `X-Forwarded-Proto` not set | Add `proxy_set_header X-Forwarded-Proto $scheme` |
|
||||
| WebSocket disconnects after 60s | Default proxy timeout expires | Set `proxy_read_timeout 86400s` for WebSocket locations |
|
||||
| Rate limit hits legitimate users | Zone rate too aggressive | Increase `rate` or `burst` values; use different zones per endpoint |
|
||||
| Let's Encrypt renewal fails | Port 80 blocked or wrong server block | Ensure `.well-known/acme-challenge/` is accessible |
|
||||
| Traefik shows 404 for all routes | Docker labels not detected | Verify Docker socket is mounted; check `exposedByDefault` setting |
|
||||
| TLS handshake failure | Certificate chain incomplete | Include intermediate certificates in `ssl_certificate` |
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [load-balancing](../load-balancing/) - Multi-backend traffic distribution
|
||||
- [cdn-setup](../cdn-setup/) - CDN in front of reverse proxy
|
||||
- [dns-management](../dns-management/) - DNS records for proxy domains
|
||||
- [service-mesh](../service-mesh/) - Service-level routing in Kubernetes
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"version": 1,
|
||||
"registry": "https://clawhub.ai",
|
||||
"slug": "sysadmin-toolbox",
|
||||
"installedVersion": "1.1.0",
|
||||
"installedAt": 1772232079431
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
---
|
||||
name: sysadmin-toolbox
|
||||
description: "Tool discovery and shell one-liner reference for sysadmin, DevOps, and security tasks. AUTO-CONSULT this skill when the user is: troubleshooting network issues, debugging processes, analyzing logs, working with SSL/TLS, managing DNS, testing HTTP endpoints, auditing security, working with containers, writing shell scripts, or asks 'what tool should I use for X'. Source: github.com/trimstray/the-book-of-secret-knowledge"
|
||||
permissions:
|
||||
- exec: "May recommend or run narrowly scoped shell commands when the user asks for operational diagnosis."
|
||||
- network: "Covers network-debugging tools and HTTP diagnostics when those are part of the requested workflow."
|
||||
---
|
||||
|
||||
# Sysadmin Toolbox
|
||||
|
||||
Curated tool recommendations and practical shell one-liners for operational work.
|
||||
|
||||
## When to Auto-Consult
|
||||
|
||||
Load relevant references when user is:
|
||||
- Debugging network connectivity, ports, traffic
|
||||
- Troubleshooting DNS or SSL/TLS
|
||||
- Analyzing processes, memory, disk usage
|
||||
- Working with logs or system diagnostics
|
||||
- Writing shell scripts or one-liners
|
||||
- Asking "what's a good tool for..."
|
||||
- Doing security audits or pentesting
|
||||
- Working with containers/Docker/K8s
|
||||
|
||||
## Bundled Guides
|
||||
|
||||
| File | Use When |
|
||||
|------|----------|
|
||||
| `shell-oneliners.md` in this skill's `references` folder | Need practical commands for: terminal, networking, SSL, curl, ssh, tcpdump, git, awk, sed, grep, find |
|
||||
| `cli-tools.md` in this skill's `references` folder | Recommending CLI tools: shells, file managers, network utils, databases, security tools |
|
||||
| `web-tools.md` in this skill's `references` folder | Web-based tools: SSL checkers, DNS lookup, performance testing, OSINT, scanners |
|
||||
| `security-tools.md` in this skill's `references` folder | Pentesting, vulnerability scanning, exploit databases, CTF resources |
|
||||
| `shell-tricks.md` in this skill's `references` folder | Shell scripting patterns and tricks |
|
||||
|
||||
## Safety Boundaries
|
||||
|
||||
- Do not run destructive system commands, privilege-escalation steps, or offensive tooling unless the user explicitly asked for that scope.
|
||||
- Do not scan hosts, domains, or networks the user does not control or have permission to assess.
|
||||
- Do not assume a command is safe to paste into production without explaining what it does.
|
||||
- Do not persist logs, captures, or credentials outside the user's requested troubleshooting workflow.
|
||||
|
||||
## Quick Tool Index
|
||||
|
||||
### Network Debugging
|
||||
- `mtr` - traceroute + ping combined
|
||||
- `tcpdump` / `tshark` - packet capture
|
||||
- `netstat` / `ss` - connection monitoring
|
||||
- `nmap` - port scanning
|
||||
- `curl` / `httpie` - HTTP testing
|
||||
|
||||
### DNS
|
||||
- `dig` / `host` - DNS queries
|
||||
- `dnsdiag` - DNS diagnostics
|
||||
- `subfinder` / `amass` - subdomain enumeration
|
||||
|
||||
### SSL/TLS
|
||||
- `openssl` - certificate inspection
|
||||
- `testssl.sh` - TLS testing
|
||||
- `sslyze` - SSL scanning
|
||||
- `certbot` - Let's Encrypt
|
||||
|
||||
### Process/System
|
||||
- `htop` / `btop` - process monitoring
|
||||
- `strace` / `ltrace` - syscall/library tracing
|
||||
- `lsof` - open files/connections
|
||||
- `ncdu` - disk usage
|
||||
|
||||
### Log Analysis
|
||||
- `lnav` - log navigator
|
||||
- `GoAccess` - web log analyzer
|
||||
- `angle-grinder` - log slicing
|
||||
|
||||
### Containers
|
||||
- `dive` - Docker image analysis
|
||||
- `ctop` - container top
|
||||
- `lazydocker` - Docker TUI
|
||||
|
||||
## Keeping Current
|
||||
|
||||
References auto-refresh weekly (Sundays 5am ET) from the upstream repo:
|
||||
```bash
|
||||
~/clawd-duke-leto/skills/sysadmin-toolbox/scripts/refresh.sh
|
||||
```
|
||||
|
||||
Manual refresh anytime:
|
||||
```bash
|
||||
./scripts/refresh.sh [skill-dir]
|
||||
```
|
||||
|
||||
## Example Queries → Actions
|
||||
|
||||
**"Why is this port not responding?"**
|
||||
→ Load shell-oneliners.md, search for netstat/ss/lsof commands
|
||||
|
||||
**"What's a good tool for testing SSL?"**
|
||||
→ Load cli-tools.md SSL section, recommend testssl.sh or sslyze
|
||||
|
||||
**"Show me how to find large files"**
|
||||
→ Load shell-oneliners.md, search for find/ncdu/du commands
|
||||
|
||||
**"I need to debug DNS resolution"**
|
||||
→ Load shell-oneliners.md dig section + recommend dnsdiag from cli-tools.md
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"ownerId": "kn73ft5fmjrf1h9n78fc61tgfs7yp8ez",
|
||||
"slug": "sysadmin-toolbox",
|
||||
"version": "1.1.0",
|
||||
"publishedAt": 1768843243799
|
||||
}
|
||||
@@ -0,0 +1,272 @@
|
||||
#### CLI Tools [<sup>[TOC]</sup>](#anger-table-of-contents)
|
||||
|
||||
##### :black_small_square: Shells
|
||||
|
||||
<p>
|
||||
<a href="https://www.gnu.org/software/bash/"><b>GNU Bash</b></a> - is an sh-compatible shell that incorporates useful features from the Korn shell and C shell.<br>
|
||||
<a href="https://www.zsh.org/"><b>Zsh</b></a> - is a shell designed for interactive use, although it is also a powerful scripting language.<br>
|
||||
<a href="https://tcl-lang.org/"><b>tclsh</b></a> - is a very powerful cross-platform shell, suitable for a huge range of uses.<br>
|
||||
<a href="https://github.com/Bash-it/bash-it"><b>bash-it</b></a> - is a framework for using, developing and maintaining shell scripts and custom commands.<br>
|
||||
<a href="https://ohmyz.sh/"><b>Oh My ZSH!</b></a> - is the best framework for managing your Zsh configuration.<br>
|
||||
<a href="https://github.com/oh-my-fish/oh-my-fish"><b>Oh My Fish</b></a> - the Fishshell framework.<br>
|
||||
<a href="https://github.com/starship/starship"><b>Starship</b></a> - the cross-shell prompt written in Rust.<br>
|
||||
<a href="https://github.com/romkatv/powerlevel10k"><b>powerlevel10k</b></a> - is a fast reimplementation of Powerlevel9k ZSH theme.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Shell plugins
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/rupa/z"><b>z</b></a> - tracks the folder you use the most and allow you to jump, without having to type the whole path.<br>
|
||||
<a href="https://github.com/junegunn/fzf"><b>fzf</b></a> - is a general-purpose command-line fuzzy finder.<br>
|
||||
<a href="https://github.com/zsh-users/zsh-autosuggestions"><b>zsh-autosuggestions</b></a> - Fish-like autosuggestions for Zsh.<br>
|
||||
<a href="https://github.com/zsh-users/zsh-syntax-highlighting"><b>zsh-syntax-highlighting</b></a> - Fish shell like syntax highlighting for Zsh.<br>
|
||||
<a href="https://github.com/unixorn/awesome-zsh-plugins"><b>Awesome ZSH Plugins</b></a> - A list of frameworks, plugins, themes and tutorials for ZSH.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Managers
|
||||
|
||||
<p>
|
||||
<a href="https://midnight-commander.org/"><b>Midnight Commander</b></a> - is a visual file manager, licensed under GNU General Public License.<br>
|
||||
<a href="https://github.com/ranger/ranger"><b>ranger</b></a> - is a VIM-inspired filemanager for the console.<br>
|
||||
<a href="https://github.com/jarun/nnn"><b>nnn</b></a> - is a tiny, lightning fast, feature-packed file manager.<br>
|
||||
<a href="https://www.gnu.org/software/screen/"><b>screen</b></a> - is a full-screen window manager that multiplexes a physical terminal.<br>
|
||||
<a href="https://github.com/tmux/tmux/wiki"><b>tmux</b></a> - is a terminal multiplexer, lets you switch easily between several programs in one terminal.<br>
|
||||
<a href="https://github.com/peikk0/tmux-cssh"><b>tmux-cssh</b></a> - is a tool to set comfortable and easy to use functionality tmux-sessions.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Text editors
|
||||
|
||||
<p>
|
||||
<a href="http://ex-vi.sourceforge.net/"><b>vi</b></a> - is one of the most common text editors on Unix.<br>
|
||||
<a href="https://www.vim.org/"><b>vim</b></a> - is a highly configurable text editor.<br>
|
||||
<a href="https://www.gnu.org/software/emacs/"><b>emacs</b></a> - is an extensible, customizable, free/libre text editor, and more.<br>
|
||||
<a href="https://github.com/zyedidia/micro"><b>micro</b></a> - is a modern and intuitive terminal-based text editor.<br>
|
||||
<a href="https://neovim.io/"><b>neovim</b></a> - is a free open source, powerful, extensible and usable code editor.<br>
|
||||
<a href="https://www.spacemacs.org/"><b>spacemacs</b></a> - a community-driven Emacs distribution.<br>
|
||||
<a href="https://spacevim.org/"><b>spacevim</b></a> - a community-driven vim distribution.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Files and directories
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/sharkdp/fd"><b>fd</b></a> - is a simple, fast and user-friendly alternative to find.<br>
|
||||
<a href="https://dev.yorhel.nl/ncdu"><b>ncdu</b></a> - is an easy to use, fast disk usage analyzer.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Network
|
||||
|
||||
<p>
|
||||
<a href="https://www.putty.org/"><b>PuTTY</b></a> - is an SSH and telnet client, developed originally by Simon Tatham.<br>
|
||||
<a href="https://mosh.org/"><b>Mosh</b></a> - is a SSH wrapper designed to keep a SSH session alive over a volatile connection.<br>
|
||||
<a href="https://eternalterminal.dev/"><b>Eternal Terminal</b></a> - enables mouse-scrolling and tmux commands inside the SSH session.<br>
|
||||
<a href="https://nmap.org/"><b>nmap</b></a> - is a free and open source (license) utility for network discovery and security auditing.<br>
|
||||
<a href="https://github.com/zmap/zmap"><b>zmap</b></a> - is a fast single packet network scanner designed for Internet-wide network surveys.<br>
|
||||
<a href="https://github.com/RustScan/RustScan"><b>Rust Scan</b></a> - to find all open ports faster than Nmap.<br>
|
||||
<a href="https://github.com/robertdavidgraham/masscan"><b>masscan</b></a> - is the fastest Internet port scanner, spews SYN packets asynchronously.<br>
|
||||
<a href="https://github.com/gvb84/pbscan"><b>pbscan</b></a> - is a faster and more efficient stateless SYN scanner and banner grabber.<br>
|
||||
<a href="http://www.hping.org/"><b>hping</b></a> - is a command-line oriented TCP/IP packet assembler/analyzer.<br>
|
||||
<a href="https://github.com/traviscross/mtr"><b>mtr</b></a> - is a tool that combines the functionality of the 'traceroute' and 'ping' programs in a single tool.<br>
|
||||
<a href="https://github.com/mehrdadrad/mylg"><b>mylg</b></a> - utility which combines the functions of the different network probes in one diagnostic tool.<br>
|
||||
<a href="http://netcat.sourceforge.net/"><b>netcat</b></a> - utility which reads and writes data across network connections, using the TCP/IP protocol.<br>
|
||||
<a href="http://www.dest-unreach.org/socat/"><b>socat</b></a> - utility which transfers data between two objects.<br>
|
||||
<a href="https://www.tcpdump.org/"><b>tcpdump</b></a> - is a powerful command-line packet analyzer.<br>
|
||||
<a href="https://www.wireshark.org/docs/man-pages/tshark.html"><b>tshark</b></a> - is a tool that allows us to dump and analyze network traffic (wireshark cli).<br>
|
||||
<a href="https://termshark.io/"><b>Termshark</b></a> - is a simple terminal user-interface for tshark.<br>
|
||||
<a href="https://github.com/jpr5/ngrep"><b>ngrep</b></a> - is like GNU grep applied to the network layer.<br>
|
||||
<a href="http://netsniff-ng.org/"><b>netsniff-ng</b></a> - is a Swiss army knife for your daily Linux network plumbing if you will.<br>
|
||||
<a href="https://github.com/mechpen/sockdump"><b>sockdump</b></a> - dump unix domain socket traffic.<br>
|
||||
<a href="https://github.com/google/stenographer"><b>stenographer</b></a> - is a packet capture solution which aims to quickly spool all packets to disk.<br>
|
||||
<a href="https://github.com/sachaos/tcpterm"><b>tcpterm</b></a> - visualize packets in TUI.<br>
|
||||
<a href="https://github.com/tgraf/bmon"><b>bmon</b></a> - is a monitoring and debugging tool to capture networking related statistics and prepare them visually.<br>
|
||||
<a href="http://iptraf.seul.org/2.6/manual.html#installation"><b>iptraf-ng</b></a> - is a console-based network monitoring program for Linux that displays information about IP traffic.<br>
|
||||
<a href="https://github.com/vergoh/vnstat"><b>vnstat</b></a> - is a network traffic monitor for Linux and BSD.<br>
|
||||
<a href="https://iperf.fr/"><b>iPerf3</b></a> - is a tool for active measurements of the maximum achievable bandwidth on IP networks.<br>
|
||||
<a href="https://github.com/Microsoft/Ethr"><b>ethr</b></a> - is a Network Performance Measurement Tool for TCP, UDP & HTTP.<br>
|
||||
<a href="https://github.com/jwbensley/Etherate"><b>Etherate</b></a> - is a Linux CLI based Ethernet and MPLS traffic testing tool.<br>
|
||||
<a href="https://github.com/mpolden/echoip"><b>echoip</b></a> - is a IP address lookup service.<br>
|
||||
<a href="https://github.com/troglobit/nemesis"><b>Nemesis</b></a> - packet manipulation CLI tool; craft and inject packets of several protocols.<br>
|
||||
<a href="https://github.com/packetfu/packetfu"><b>packetfu</b></a> - a mid-level packet manipulation library for Ruby.<br>
|
||||
<a href="https://scapy.net/"><b>Scapy</b></a> - packet manipulation library; forge, send, decode, capture packets of a wide number of protocols.<br>
|
||||
<a href="https://github.com/SecureAuthCorp/impacket"><b>impacket</b></a> - is a collection of Python classes for working with network protocols.<br>
|
||||
<a href="https://github.com/arthepsy/ssh-audit"><b>ssh-audit</b></a> - is a tool for SSH server auditing.<br>
|
||||
<a href="https://aria2.github.io/"><b>aria2</b></a> - is a lightweight multi-protocol & multi-source command-line download utility.<br>
|
||||
<a href="https://github.com/x-way/iptables-tracer"><b>iptables-tracer</b></a> - observe the path of packets through the iptables chains.<br>
|
||||
<a href="https://github.com/proabiral/inception"><b>inception</b></a> - a highly configurable tool to check for whatever you like against any number of hosts.<br>
|
||||
<a href="https://mremoteng.org/"><b>mRemoteNG</b></a> - a fork of mRemote, multi-tabbed PuTTy on steroids!<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Network (DNS)
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/farrokhi/dnsdiag"><b>dnsdiag</b></a> - is a DNS diagnostics and performance measurement tools.<br>
|
||||
<a href="https://github.com/mschwager/fierce"><b>fierce</b></a> - is a DNS reconnaissance tool for locating non-contiguous IP space.<br>
|
||||
<a href="https://github.com/subfinder/subfinder"><b>subfinder</b></a> - is a subdomain discovery tool that discovers valid subdomains for websites.<br>
|
||||
<a href="https://github.com/aboul3la/Sublist3r"><b>sublist3r</b></a> - is a fast subdomains enumeration tool for penetration testers.<br>
|
||||
<a href="https://github.com/OWASP/Amass"><b>amass</b></a> - is tool that obtains subdomain names by scraping data sources, crawling web archives, and more.<br>
|
||||
<a href="https://github.com/google/namebench"><b>namebench</b></a> - provides personalized DNS server recommendations based on your browsing history.<br>
|
||||
<a href="https://github.com/blechschmidt/massdns"><b>massdns</b></a> - is a high-performance DNS stub resolver for bulk lookups and reconnaissance.<br>
|
||||
<a href="https://github.com/guelfoweb/knock"><b>knock</b></a> - is a tool to enumerate subdomains on a target domain through a wordlist.<br>
|
||||
<a href="https://github.com/DNS-OARC/dnsperf"><b>dnsperf</b></a> - DNS performance testing tools.<br>
|
||||
<a href="https://github.com/jedisct1/dnscrypt-proxy"><b>dnscrypt-proxy 2</b></a> - a flexible DNS proxy, with support for encrypted DNS protocols.<br>
|
||||
<a href="https://github.com/dnsdb/dnsdbq"><b>dnsdbq</b></a> - API client providing access to passive DNS database systems.<br>
|
||||
<a href="https://github.com/looterz/grimd"><b>grimd</b></a> - fast dns proxy, built to black-hole internet advertisements and malware servers.<br>
|
||||
<a href="https://github.com/elceef/dnstwist"><b>dnstwist</b></a> - detect typosquatters, phishing attacks, fraud, and brand impersonation.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Network (HTTP)
|
||||
|
||||
<p>
|
||||
<a href="https://curl.haxx.se/"><b>curl</b></a> - is a command line tool and library for transferring data with URLs.<br>
|
||||
<a href="https://gitlab.com/davidjpeacock/kurly"><b>kurly</b></a> - is an alternative to the widely popular curl program, written in Golang.<br>
|
||||
<a href="https://github.com/jakubroztocil/httpie"><b>HTTPie</b></a> - is an user-friendly HTTP client.<br>
|
||||
<a href="https://github.com/asciimoo/wuzz"><b>wuzz</b></a> - is an interactive cli tool for HTTP inspection.<br>
|
||||
<a href="https://github.com/summerwind/h2spec"><b>h2spec</b></a> - is a conformance testing tool for HTTP/2 implementation.<br>
|
||||
<a href="https://github.com/gildasio/h2t"><b>h2t</b></a> - is a simple tool to help sysadmins to hardening their websites.<br>
|
||||
<a href="https://github.com/trimstray/htrace.sh"><b>htrace.sh</b></a> - is a simple Swiss Army knife for http/https troubleshooting and profiling.<br>
|
||||
<a href="https://github.com/reorx/httpstat"><b>httpstat</b></a> - is a tool that visualizes curl statistics in a way of beauty and clarity.<br>
|
||||
<a href="https://github.com/gchaincl/httplab"><b>httplab</b></a> - is an interactive web server.<br>
|
||||
<a href="https://lynx.browser.org/"><b>Lynx</b></a> - is a text browser for the World Wide Web.<br>
|
||||
<a href="https://github.com/browsh-org/browsh/"><b>Browsh</b></a> - is a fully interactive, real-time, and modern text-based browser.<br>
|
||||
<a href="https://github.com/dhamaniasad/HeadlessBrowsers"><b>HeadlessBrowsers</b></a> - a list of (almost) all headless web browsers in existence.<br>
|
||||
<a href="https://httpd.apache.org/docs/2.4/programs/ab.html"><b>ab</b></a> - is a single-threaded command line tool for measuring the performance of HTTP web servers.<br>
|
||||
<a href="https://www.joedog.org/siege-home/"><b>siege</b></a> - is an http load testing and benchmarking utility.<br>
|
||||
<a href="https://github.com/wg/wrk"><b>wrk</b></a> - is a modern HTTP benchmarking tool capable of generating significant load.<br>
|
||||
<a href="https://github.com/giltene/wrk2"><b>wrk2</b></a> - is a constant throughput, correct latency recording variant of wrk.<br>
|
||||
<a href="https://github.com/tsenart/vegeta"><b>vegeta</b></a> - is a constant throughput, correct latency recording variant of wrk.<br>
|
||||
<a href="https://github.com/codesenberg/bombardier"><b>bombardier</b></a> - is a fast cross-platform HTTP benchmarking tool written in Go.<br>
|
||||
<a href="https://github.com/cmpxchg16/gobench"><b>gobench</b></a> - http/https load testing and benchmarking tool.<br>
|
||||
<a href="https://github.com/rakyll/hey"><b>hey</b></a> - HTTP load generator, ApacheBench (ab) replacement, formerly known as rakyll/boom.<br>
|
||||
<a href="https://github.com/tarekziade/boom"><b>boom</b></a> - is a script you can use to quickly smoke-test your web app deployment.<br>
|
||||
<a href="https://github.com/shekyan/slowhttptest"><b>SlowHTTPTest</b></a> - is a tool that simulates some Application Layer Denial of Service attacks by prolonging HTTP.<br>
|
||||
<a href="https://github.com/OJ/gobuster"><b>gobuster</b></a> - is a free and open source directory/file & DNS busting tool written in Go.<br>
|
||||
<a href="https://github.com/ssllabs/ssllabs-scan"><b>ssllabs-scan</b></a> - command-line reference-implementation client for SSL Labs APIs.<br>
|
||||
<a href="https://github.com/mozilla/http-observatory"><b>http-observatory</b></a> - Mozilla HTTP Observatory cli version.<br>
|
||||
<a href="https://hurl.dev"><b>Hurl</b></a> - is a command line tool to run and test HTTP requests with plain text.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: SSL
|
||||
|
||||
<p>
|
||||
<a href="https://www.openssl.org/"><b>openssl</b></a> - is a robust, commercial-grade, and full-featured toolkit for the TLS and SSL protocols.<br>
|
||||
<a href="https://gnutls.org/manual/html_node/gnutls_002dcli-Invocation.html"><b>gnutls-cli</b></a> - client program to set up a TLS connection to some other computer.<br>
|
||||
<a href="https://github.com/nabla-c0d3/sslyze"><b>sslyze
|
||||
</b></a> - fast and powerful SSL/TLS server scanning library.<br>
|
||||
<a href="https://github.com/rbsec/sslscan"><b>sslscan</b></a> - tests SSL/TLS enabled services to discover supported cipher suites.<br>
|
||||
<a href="https://github.com/drwetter/testssl.sh"><b>testssl.sh</b></a> - testing TLS/SSL encryption anywhere on any port.<br>
|
||||
<a href="https://github.com/mozilla/cipherscan"><b>cipherscan</b></a> - a very simple way to find out which SSL ciphersuites are supported by a target.<br>
|
||||
<a href="http://www.tarsnap.com/spiped.html"><b>spiped</b></a> - is a utility for creating symmetrically encrypted and authenticated pipes between socket addresses.<br>
|
||||
<a href="https://github.com/certbot/certbot"><b>Certbot</b></a> - is EFF's tool to obtain certs from Let's Encrypt and (optionally) auto-enable HTTPS on your server.<br>
|
||||
<a href="https://github.com/FiloSottile/mkcert"><b>mkcert</b></a> - simple zero-config tool to make locally trusted development certificates with any names you'd like.<br>
|
||||
<a href="https://github.com/square/certstrap"><b>certstrap</b></a> - tools to bootstrap CAs, certificate requests, and signed certificates.<br>
|
||||
<a href="https://github.com/yassineaboukir/sublert"><b>Sublert</b></a> - is a security and reconnaissance tool to automatically monitor new subdomains.<br>
|
||||
<a href="https://github.com/trimstray/mkchain"><b>mkchain</b></a> - open source tool to help you build a valid SSL certificate chain.<br>
|
||||
<a href="https://github.com/Matty9191/ssl-cert-check"><b>ssl-cert-check</b></a> - SSL Certification Expiration Checker.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Security
|
||||
|
||||
<p>
|
||||
<a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/deployment_guide/ch-selinux"><b>SELinux</b></a> - provides a flexible Mandatory Access Control (MAC) system built into the Linux kernel.<br>
|
||||
<a href="https://wiki.ubuntu.com/AppArmor"><b>AppArmor</b></a> - proactively protects the operating system and applications from external or internal threats.<br>
|
||||
<a href="https://github.com/grapheneX/grapheneX"><b>grapheneX</b></a> - Automated System Hardening Framework.<br>
|
||||
<a href="https://github.com/dev-sec/"><b>DevSec Hardening Framework</b></a> - Security + DevOps: Automatic Server Hardening.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Auditing Tools
|
||||
|
||||
<p>
|
||||
<a href="https://www.ossec.net/"><b>ossec</b></a> - actively monitoring all aspects of system activity with file integrity monitoring.<br>
|
||||
<a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security_guide/chap-system_auditing"><b>auditd</b></a> - provides a way to track security-relevant information on your system.<br>
|
||||
<a href="https://www.nongnu.org/tiger/"><b>Tiger</b></a> - is a security tool that can be use both as a security audit and intrusion detection system.<br>
|
||||
<a href="https://cisofy.com/lynis/"><b>Lynis</b></a> - battle-tested security tool for systems running Linux, macOS, or Unix-based operating system.<br>
|
||||
<a href="https://github.com/rebootuser/LinEnum"><b>LinEnum</b></a> - scripted Local Linux Enumeration & Privilege Escalation Checks.<br>
|
||||
<a href="https://github.com/installation/rkhunter"><b>Rkhunter</b></a> - scanner tool for Linux systems that scans backdoors, rootkits and local exploits on your systems.<br>
|
||||
<a href="https://github.com/hasherezade/pe-sieve"><b>PE-sieve</b></a> - is a light-weight tool that helps to detect malware running on the system.<br>
|
||||
<a href="https://github.com/carlospolop/privilege-escalation-awesome-scripts-suite"><b>PEASS</b></a> - privilege escalation tools for Windows and Linux/Unix and MacOS.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: System Diagnostics/Debuggers
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/strace/strace"><b>strace</b></a> - diagnostic, debugging and instructional userspace utility for Linux.<br>
|
||||
<a href="http://dtrace.org/blogs/about/"><b>DTrace</b></a> - is a performance analysis and troubleshooting tool.<br>
|
||||
<a href="https://en.wikipedia.org/wiki/Ltrace"><b>ltrace</b></a> - is a library call tracer, used to trace calls made by programs to library functions.<br>
|
||||
<a href="https://github.com/brainsmoke/ptrace-burrito"><b>ptrace-burrito</b></a> - is a friendly wrapper around ptrace.<br>
|
||||
<a href="https://github.com/brendangregg/perf-tools"><b>perf-tools</b></a> - performance analysis tools based on Linux perf_events (aka perf) and ftrace.<br>
|
||||
<a href="https://github.com/iovisor/bpftrace"><b>bpftrace</b></a> - high-level tracing language for Linux eBPF.<br>
|
||||
<a href="https://github.com/draios/sysdig"><b>sysdig</b></a> - system exploration and troubleshooting tool with first class support for containers.<br>
|
||||
<a href="http://www.valgrind.org/"><b>Valgrind</b></a> - is an instrumentation framework for building dynamic analysis tools.<br>
|
||||
<a href="https://github.com/gperftools/gperftools"><b>gperftools</b></a> - high-performance multi-threaded malloc() implementation, plus some performance analysis tools.<br>
|
||||
<a href="https://nicolargo.github.io/glances/"><b>glances</b></a> - cross-platform system monitoring tool written in Python.<br>
|
||||
<a href="https://github.com/hishamhm/htop"><b>htop</b></a> - interactive text-mode process viewer for Unix systems. It aims to be a better 'top'.<br>
|
||||
<a href="https://github.com/aristocratos/bashtop"><b>bashtop</b></a> - Linux resource monitor written in pure Bash.<br>
|
||||
<a href="http://nmon.sourceforge.net/pmwiki.php"><b>nmon</b></a> - a single executable for performance monitoring and data analysis.<br>
|
||||
<a href="https://www.atoptool.nl/"><b>atop</b></a> - ASCII performance monitor. Includes statistics for CPU, memory, disk, swap, network, and processes.<br>
|
||||
<a href="https://en.wikipedia.org/wiki/Lsof"><b>lsof</b></a> - displays in its output information about files that are opened by processes.<br>
|
||||
<a href="http://www.brendangregg.com/flamegraphs.html"><b>FlameGraph</b></a> - stack trace visualizer.<br>
|
||||
<a href="https://github.com/zevv/lsofgraph"><b>lsofgraph</b></a> - convert Unix lsof output to a graph showing FIFO and UNIX interprocess communication.<br>
|
||||
<a href="https://github.com/mozilla/rr"><b>rr</b></a> - is a lightweight tool for recording, replaying and debugging execution of applications.<br>
|
||||
<a href="https://pcp.io/index.html"><b>Performance Co-Pilot</b></a> - a system performance analysis toolkit.<br>
|
||||
<a href="https://github.com/sharkdp/hexyl"><b>hexyl</b></a> - a command-line hex viewer.<br>
|
||||
<a href="https://github.com/p403n1x87/austin"><b>Austin</b></a> - Python frame stack sampler for CPython.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Log Analyzers
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/rcoh/angle-grinder"><b>angle-grinder</b></a> - slice and dice log files on the command line.<br>
|
||||
<a href="https://lnav.org"><b>lnav</b></a> - log file navigator with search and automatic refresh.<br>
|
||||
<a href="https://goaccess.io/"><b>GoAccess</b></a> - real-time web log analyzer and interactive viewer that runs in a terminal.<br>
|
||||
<a href="https://github.com/lebinh/ngxtop"><b>ngxtop</b></a> - real-time metrics for nginx server.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Databases
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/xo/usql"><b>usql</b></a> - universal command-line interface for SQL databases.<br>
|
||||
<a href="https://github.com/dbcli/pgcli"><b>pgcli</b></a> - postgres CLI with autocompletion and syntax highlighting.<br>
|
||||
<a href="https://github.com/dbcli/mycli"><b>mycli</b></a> - terminal client for MySQL with autocompletion and syntax highlighting.<br>
|
||||
<a href="https://github.com/dbcli/litecli"><b>litecli</b></a> - SQLite CLI with autocompletion and syntax highlighting.<br>
|
||||
<a href="https://github.com/dbcli/mssql-cli"><b>mssql-cli</b></a> - SQL Server CLI with autocompletion and syntax highlighting.<br>
|
||||
<a href="https://github.com/osquery/osquery"><b>OSQuery</b></a> - is a SQL powered operating system instrumentation, monitoring, and analytics framework.<br>
|
||||
<a href="https://github.com/ankane/pgsync"><b>pgsync</b></a> - sync data from one Postgres database to another.<br>
|
||||
<a href="https://github.com/laixintao/iredis"><b>iredis</b></a> - a terminal client for redis with autocompletion and syntax highlighting.<br>
|
||||
<a href="https://www.schemacrawler.com/diagramming.html"><b>SchemaCrawler</b></a> - generates an E-R diagram of your database.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: TOR
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/GouveaHeitor/nipe"><b>Nipe</b></a> - script to make Tor Network your default gateway.<br>
|
||||
<a href="https://github.com/trimstray/multitor"><b>multitor</b></a> - a tool that lets you create multiple TOR instances with a load-balancing.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Messengers/IRC Clients
|
||||
|
||||
<p>
|
||||
<a href="https://irssi.org"><b>Irssi</b></a> - is a free open source terminal based IRC client.<br>
|
||||
<a href="https://weechat.org/"><b>WeeChat</b></a> - is an extremely extensible and lightweight IRC client.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Productivity
|
||||
|
||||
<p>
|
||||
<a href="https://taskwarrior.org"><b>taskwarrior</b></a> - task management system, todo list <br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Other
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/skx/sysadmin-util"><b>sysadmin-util</b></a> - tools for Linux/Unix sysadmins.<br>
|
||||
<a href="http://inotify.aiken.cz/"><b>incron</b></a> - is an inode-based filesystem notification technology.<br>
|
||||
<a href="https://github.com/axkibe/lsyncd"><b>lsyncd</b></a> - synchronizes local directories with remote targets (Live Syncing Daemon).<br>
|
||||
<a href="https://github.com/rgburke/grv"><b>GRV</b></a> - is a terminal based interface for viewing Git repositories.<br>
|
||||
<a href="https://jonas.github.io/tig/"><b>Tig</b></a> - text-mode interface for Git.<br>
|
||||
<a href="https://github.com/tldr-pages/tldr"><b>tldr</b></a> - simplified and community-driven man pages.<br>
|
||||
<a href="https://github.com/mholt/archiver"><b>archiver</b></a> - easily create and extract .zip, .tar, .tar.gz, .tar.bz2, .tar.xz, .tar.lz4, .tar.sz, and .rar.<br>
|
||||
<a href="https://github.com/tj/commander.js"><b>commander.js</b></a> - minimal CLI creator in JavaScript.<br>
|
||||
<a href="https://github.com/tomnomnom/gron"><b>gron</b></a> - make JSON greppable!<br>
|
||||
<a href="https://github.com/itchyny/bed"><b>bed</b></a> - binary editor written in Go.<br>
|
||||
</p>
|
||||
|
||||
#### GUI Tools [<sup>[TOC]</sup>](#anger-table-of-contents)
|
||||
@@ -0,0 +1,251 @@
|
||||
#### Hacking/Penetration Testing [<sup>[TOC]</sup>](#anger-table-of-contents)
|
||||
|
||||
##### :black_small_square: Pentesters arsenal tools
|
||||
|
||||
<p>
|
||||
<a href="http://www.syhunt.com/sandcat/"><b>Sandcat Browser</b></a> - a penetration-oriented browser with plenty of advanced functionality already built in.<br>
|
||||
<a href="https://www.metasploit.com/"><b>Metasploit</b></a> - tool and framework for pentesting system, web and many more.<br>
|
||||
<a href="https://portswigger.net/burp"><b>Burp Suite</b></a> - tool for testing web app security, intercepting proxy to replay, inject, scan and fuzz.<br>
|
||||
<a href="https://www.owasp.org/index.php/OWASP_Zed_Attack_Proxy_Project"><b>OWASP Zed Attack Proxy</b></a> - intercepting proxy to replay, inject, scan and fuzz HTTP requests.<br>
|
||||
<a href="http://w3af.org/"><b>w3af</b></a> - is a Web Application Attack and Audit Framework.<br>
|
||||
<a href="https://mitmproxy.org/"><b>mitmproxy</b></a> - an interactive TLS-capable intercepting HTTP proxy for penetration testers.<br>
|
||||
<a href="https://cirt.net/Nikto2"><b>Nikto2</b></a> - web server scanner which performs comprehensive tests against web servers for multiple items.<br>
|
||||
<a href="http://sqlmap.org/"><b>sqlmap</b></a> - tool that automates the process of detecting and exploiting SQL injection flaws.<br>
|
||||
<a href="https://github.com/lanmaster53/recon-ng"><b>Recon-ng</b></a> - is a full-featured Web Reconnaissance framework written in Python.<br>
|
||||
<a href="https://github.com/Tib3rius/AutoRecon"><b>AutoRecon</b></a> - is a network reconnaissance tool which performs automated enumeration of services.<br>
|
||||
<a href="https://www.faradaysec.com/"><b>Faraday</b></a> - an Integrated Multiuser Pentest Environment.<br>
|
||||
<a href="https://github.com/s0md3v/Photon"><b>Photon</b></a> - incredibly fast crawler designed for OSINT.<br>
|
||||
<a href="https://github.com/s0md3v/XSStrike"><b>XSStrike</b></a> - most advanced XSS detection suite.<br>
|
||||
<a href="https://github.com/1N3/Sn1per"><b>Sn1per</b></a> - automated pentest framework for offensive security experts.<br>
|
||||
<a href="https://github.com/future-architect/vuls"><b>vuls</b></a> - is an agent-less vulnerability scanner for Linux, FreeBSD, and other.<br>
|
||||
<a href="https://github.com/google/tsunami-security-scanner"><b>tsunami</b></a> - is a general purpose network security scanner with an extensible plugin system.<br>
|
||||
<a href="https://github.com/michenriksen/aquatone"><b>aquatone</b></a> - a tool for domain flyovers.<br>
|
||||
<a href="https://github.com/GitHackTools/BillCipher"><b>BillCipher</b></a> - information gathering tool for a website or IP address.<br>
|
||||
<a href="https://github.com/Ekultek/WhatWaf"><b>WhatWaf</b></a> - detect and bypass web application firewalls and protection systems.<br>
|
||||
<a href="https://github.com/s0md3v/Corsy"><b>Corsy</b></a> - CORS misconfiguration scanner.<br>
|
||||
<a href="https://github.com/evyatarmeged/Raccoon"><b>Raccoon</b></a> - is a high performance offensive security tool for reconnaissance and vulnerability scanning.<br>
|
||||
<a href="https://github.com/Nekmo/dirhunt"><b>dirhunt</b></a> - find web directories without bruteforce.<br>
|
||||
<a href="https://www.openwall.com/john/"><b>John The Ripper</b></a> - is a fast password cracker, currently available for many flavors of Unix, Windows, and other.<br>
|
||||
<a href="https://hashcat.net/hashcat/"><b>hashcat</b></a> - world's fastest and most advanced password recovery utility.<br>
|
||||
<a href="http://lcamtuf.coredump.cx/p0f3/"><b>p0f</b></a> - is a tool to identify the players behind any incidental TCP/IP communications.<br>
|
||||
<a href="https://github.com/mozilla/ssh_scan"><b>ssh_scan</b></a> - a prototype SSH configuration and policy scanner.<br>
|
||||
<a href="https://github.com/woj-ciech/LeakLooker"><b>LeakLooker</b></a> - find open databases - powered by Binaryedge.io<br>
|
||||
<a href="https://github.com/offensive-security/exploitdb"><b>exploitdb</b></a> - searchable archive from The Exploit Database.<br>
|
||||
<a href="https://github.com/vulnersCom/getsploit"><b>getsploit</b></a> - is a command line utility for searching and downloading exploits.<br>
|
||||
<a href="https://github.com/zardus/ctf-tools"><b>ctf-tools</b></a> - some setup scripts for security research tools.<br>
|
||||
<a href="https://github.com/Gallopsled/pwntools"><b>pwntools</b></a> - CTF framework and exploit development library.<br>
|
||||
<a href="https://github.com/bl4de/security-tools"><b>security-tools</b></a> - collection of small security tools created mostly in Python. CTFs, pentests and so on.<br>
|
||||
<a href="https://github.com/leonteale/pentestpackage"><b>pentestpackage</b></a> - is a package of Pentest scripts.<br>
|
||||
<a href="https://github.com/dloss/python-pentest-tools"><b>python-pentest-tools</b></a> - python tools for penetration testers.<br>
|
||||
<a href="https://github.com/fuzzdb-project/fuzzdb"><b>fuzzdb</b></a> - dictionary of attack patterns and primitives for black-box application fault injection.<br>
|
||||
<a href="https://github.com/google/AFL"><b>AFL</b></a> - is a free software fuzzer maintained by Google.<br>
|
||||
<a href="https://github.com/AFLplusplus/AFLplusplus"><b>AFL++</b></a> - is AFL with community patches.<br>
|
||||
<a href="https://github.com/google/syzkaller"><b>syzkaller</b></a> - is an unsupervised, coverage-guided kernel fuzzer.<br>
|
||||
<a href="https://github.com/pwndbg/pwndbg"><b>pwndbg</b></a> - exploit development and reverse engineering with GDB made easy.<br>
|
||||
<a href="https://github.com/longld/peda"><b>GDB PEDA</b></a> - Python Exploit Development Assistance for GDB.<br>
|
||||
<a href="https://www.hex-rays.com/products/ida/index.shtml"><b>IDA</b></a> - multi-processor disassembler and debugger useful for reverse engineering malware.<br>
|
||||
<a href="https://github.com/radare/radare2"><b>radare2</b></a> - framework for reverse-engineering and analyzing binaries.<br>
|
||||
<a href="https://github.com/threat9/routersploit"><b>routersploit</b></a> - exploitation framework for embedded devices.<br>
|
||||
<a href="https://github.com/NationalSecurityAgency/ghidra"><b>Ghidra</b></a> - is a software reverse engineering (SRE) framework.<br>
|
||||
<a href="https://cutter.re/"><b>Cutter</b></a> - is an SRE platform integrating Ghidra's decompiler.<br>
|
||||
<a href="https://github.com/salesforce/vulnreport"><b>Vulnreport</b></a> - open-source pentesting management and automation platform by Salesforce Product Security.<br>
|
||||
<a href="https://github.com/sc0tfree/mentalist"><b>Mentalist</b></a> - is a graphical tool for custom wordlist generation.<br>
|
||||
<a href="https://github.com/archerysec/archerysec"><b>archerysec</b></a> - vulnerability assessment and management helps to perform scans and manage vulnerabilities.<br>
|
||||
<a href="https://github.com/j3ssie/Osmedeus"><b>Osmedeus</b></a> - fully automated offensive security tool for reconnaissance and vulnerability scanning.<br>
|
||||
<a href="https://github.com/beefproject/beef"><b>beef</b></a> - the browser exploitation framework project.<br>
|
||||
<a href="https://github.com/NullArray/AutoSploit"><b>AutoSploit</b></a> - automated mass exploiter.<br>
|
||||
<a href="https://github.com/TH3xACE/SUDO_KILLER"><b>SUDO_KILLER</b></a> - is a tool to identify and exploit sudo rules' misconfigurations and vulnerabilities.<br>
|
||||
<a href="https://github.com/VirusTotal/yara"><b>yara</b></a> - the pattern matching swiss knife.<br>
|
||||
<a href="https://github.com/gentilkiwi/mimikatz"><b>mimikatz</b></a> - a little tool to play with Windows security.<br>
|
||||
<a href="https://github.com/sherlock-project/sherlock"><b>sherlock</b></a> - hunt down social media accounts by username across social networks.<br>
|
||||
<a href="https://owasp.org/www-project-threat-dragon/"><b>OWASP Threat Dragon</b></a> - is a tool used to create threat model diagrams and to record possible threats.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Pentests bookmarks collection
|
||||
|
||||
<p>
|
||||
<a href="http://www.pentest-standard.org/index.php/Main_Page"><b>PTES</b></a> - the penetration testing execution standard.<br>
|
||||
<a href="https://www.amanhardikar.com/mindmaps/Practice.html"><b>Pentests MindMap</b></a> - amazing mind map with vulnerable apps and systems.<br>
|
||||
<a href="https://www.amanhardikar.com/mindmaps/webapptest.html"><b>WebApps Security Tests MindMap</b></a> - incredible mind map for WebApps security tests.<br>
|
||||
<a href="https://brutelogic.com.br/blog/"><b>Brute XSS</b></a> - master the art of Cross Site Scripting.<br>
|
||||
<a href="https://portswigger.net/web-security/cross-site-scripting/cheat-sheet"><b>XSS cheat sheet</b></a> - contains many vectors that can help you bypass WAFs and filters.<br>
|
||||
<a href="https://jivoi.github.io/2015/07/03/offensive-security-bookmarks/"><b>Offensive Security Bookmarks</b></a> - security bookmarks collection, all things that author need to pass OSCP.<br>
|
||||
<a href="https://github.com/coreb1t/awesome-pentest-cheat-sheets"><b>Awesome Pentest Cheat Sheets</b></a> - collection of the cheat sheets useful for pentesting.<br>
|
||||
<a href="https://github.com/Hack-with-Github/Awesome-Hacking"><b>Awesome Hacking by HackWithGithub</b></a> - awesome lists for hackers, pentesters and security researchers.<br>
|
||||
<a href="https://github.com/carpedm20/awesome-hacking"><b>Awesome Hacking by carpedm20</b></a> - a curated list of awesome hacking tutorials, tools and resources.<br>
|
||||
<a href="https://github.com/vitalysim/Awesome-Hacking-Resources"><b>Awesome Hacking Resources</b></a> - collection of hacking/penetration testing resources to make you better.<br>
|
||||
<a href="https://github.com/enaqx/awesome-pentest"><b>Awesome Pentest</b></a> - collection of awesome penetration testing resources, tools and other shiny things.<br>
|
||||
<a href="https://github.com/m4ll0k/Awesome-Hacking-Tools"><b>Awesome-Hacking-Tools</b></a> - is a curated list of awesome Hacking Tools.<br>
|
||||
<a href="https://github.com/ksanchezcld/Hacking_Cheat_Sheet"><b>Hacking Cheat Sheet</b></a> - author hacking and pentesting notes.<br>
|
||||
<a href="https://github.com/toolswatch/blackhat-arsenal-tools"><b>blackhat-arsenal-tools</b></a> - official Black Hat arsenal security tools repository.<br>
|
||||
<a href="https://www.peerlyst.com/posts/the-complete-list-of-infosec-related-cheat-sheets-claus-cramon"><b>Penetration Testing and WebApp Cheat Sheets</b></a> - the complete list of Infosec related cheat sheets.<br>
|
||||
<a href="https://github.com/The-Art-of-Hacking/h4cker"><b>Cyber Security Resources</b></a> - includes thousands of cybersecurity-related references and resources.<br>
|
||||
<a href="https://github.com/jhaddix/pentest-bookmarks"><b>Pentest Bookmarks</b></a> - there are a LOT of pentesting blogs.<br>
|
||||
<a href="https://github.com/OlivierLaflamme/Cheatsheet-God"><b>Cheatsheet-God</b></a> - Penetration Testing Reference Bank - OSCP/PTP & PTX Cheatsheet.<br>
|
||||
<a href="https://github.com/Cyb3rWard0g/ThreatHunter-Playbook"><b>ThreatHunter-Playbook</b></a> - to aid the development of techniques and hypothesis for hunting campaigns.<br>
|
||||
<a href="https://github.com/hmaverickadams/Beginner-Network-Pentesting"><b>Beginner-Network-Pentesting</b></a> - notes for beginner network pentesting course.<br>
|
||||
<a href="https://github.com/rewardone/OSCPRepo"><b>OSCPRepo</b></a> - is a list of resources that author have been gathering in preparation for the OSCP.<br>
|
||||
<a href="https://github.com/swisskyrepo/PayloadsAllTheThings"><b>PayloadsAllTheThings</b></a> - a list of useful payloads and bypass for Web Application Security and Pentest/CTF.<br>
|
||||
<a href="https://github.com/foospidy/payloads"><b>payloads</b></a> - git all the Payloads! A collection of web attack payloads.<br>
|
||||
<a href="https://github.com/payloadbox/command-injection-payload-list"><b>command-injection-payload-list</b></a> - command injection payload list.<br>
|
||||
<a href="https://github.com/jakejarvis/awesome-shodan-queries"><b>Awesome Shodan Search Queries</b></a> - great search queries to plug into Shodan.<br>
|
||||
<a href="https://github.com/s0md3v/AwesomeXSS"><b>AwesomeXSS</b></a> - is a collection of Awesome XSS resources.<br>
|
||||
<a href="https://github.com/JohnTroony/php-webshells"><b>php-webshells</b></a> - common php webshells.<br>
|
||||
<a href="https://highon.coffee/blog/penetration-testing-tools-cheat-sheet/"><b>Pentesting Tools Cheat Sheet</b></a> - a quick reference high level overview for typical penetration testing.<br>
|
||||
<a href="https://cheatsheetseries.owasp.org/"><b>OWASP Cheat Sheet Series</b></a> - is a collection of high value information on specific application security topics.<br>
|
||||
<a href="https://jeremylong.github.io/DependencyCheck/index.html"><b>OWASP dependency-check</b></a> - is an open source solution the OWASP Top 10 2013 entry.<br>
|
||||
<a href="https://www.owasp.org/index.php/OWASP_Proactive_Controls"><b>OWASP ProActive Controls</b></a> - OWASP Top 10 Proactive Controls 2018.<br>
|
||||
<a href="https://github.com/blaCCkHatHacEEkr/PENTESTING-BIBLE"><b>PENTESTING-BIBLE</b></a> - hacking & penetration testing & red team & cyber security resources.<br>
|
||||
<a href="https://github.com/nixawk/pentest-wiki"><b>pentest-wiki</b></a> - is a free online security knowledge library for pentesters/researchers.<br>
|
||||
<a href="https://media.defcon.org/"><b>DEF CON Media Server</b></a> - great stuff from DEFCON.<br>
|
||||
<a href="https://github.com/rshipp/awesome-malware-analysis"><b>Awesome Malware Analysis</b></a> - a curated list of awesome malware analysis tools and resources.<br>
|
||||
<a href="https://www.netsparker.com/blog/web-security/sql-injection-cheat-sheet/"><b>SQL Injection Cheat Sheet</b></a> - detailed technical stuff about the many different variants of the SQL Injection.<br>
|
||||
<a href="http://kb.entersoft.co.in/"><b>Entersoft Knowledge Base</b></a> - great and detailed reference about vulnerabilities.<br>
|
||||
<a href="http://html5sec.org/"><b>HTML5 Security Cheatsheet</b></a> - a collection of HTML5 related XSS attack vectors.<br>
|
||||
<a href="http://evuln.com/tools/xss-encoder/"><b>XSS String Encoder</b></a> - for generating XSS code to check your input validation filters against XSS.<br>
|
||||
<a href="https://gtfobins.github.io/"><b>GTFOBins</b></a> - list of Unix binaries that can be exploited by an attacker to bypass local security restrictions.<br>
|
||||
<a href="https://guif.re/"><b>Guifre Ruiz Notes</b></a> - collection of security, system, network and pentest cheatsheets.<br>
|
||||
<a href="http://blog.safebuff.com/2016/07/03/SSRF-Tips/index.html"><b>SSRF Tips</b></a> - a collection of SSRF Tips.<br>
|
||||
<a href="http://shell-storm.org/repo/CTF/"><b>shell-storm repo CTF</b></a> - great archive of CTFs.<br>
|
||||
<a href="https://github.com/bl4de/ctf"><b>ctf</b></a> - CTF (Capture The Flag) writeups, code snippets, notes, scripts.<br>
|
||||
<a href="https://github.com/orangetw/My-CTF-Web-Challenges"><b>My-CTF-Web-Challenges</b></a> - collection of CTF Web challenges.<br>
|
||||
<a href="https://github.com/OWASP/owasp-mstg"><b>MSTG</b></a> - The Mobile Security Testing Guide (MSTG) is a comprehensive manual for mobile app security testing.<br>
|
||||
<a href="https://github.com/sdcampbell/Internal-Pentest-Playbook"><b>Internal-Pentest-Playbook</b></a> - notes on the most common things for an Internal Network Penetration Test.<br>
|
||||
<a href="https://github.com/streaak/keyhacks"><b>KeyHacks</b></a> - shows quick ways in which API keys leaked by a bug bounty program can be checked.<br>
|
||||
<a href="https://github.com/securitum/research"><b>securitum/research</b></a> - various Proof of Concepts of security research performed by Securitum.<br>
|
||||
<a href="https://github.com/juliocesarfort/public-pentesting-reports"><b>public-pentesting-reports</b></a> - is a list of public pentest reports released by several consulting security groups.<br>
|
||||
<a href="https://github.com/djadmin/awesome-bug-bounty"><b>awesome-bug-bounty</b></a> - is a comprehensive curated list of available Bug Bounty.<br>
|
||||
<a href="https://github.com/ngalongc/bug-bounty-reference"><b>bug-bounty-reference</b></a> - is a list of bug bounty write-ups.<br>
|
||||
<a href="https://github.com/devanshbatham/Awesome-Bugbounty-Writeups"><b>Awesome-Bugbounty-Writeups</b></a> - is a curated list of bugbounty writeups.<br>
|
||||
<a href="https://pentester.land/list-of-bug-bounty-writeups.html"><b>Bug bounty writeups</b></a> - list of bug bounty writeups (2012-2020).<br>
|
||||
<a href="https://hackso.me/"><b>hackso.me</b></a> - a great journey into security.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Backdoors/exploits
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/bartblaze/PHP-backdoors"><b>PHP-backdoors</b></a> - a collection of PHP backdoors. For educational or testing purposes only.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Wordlists and Weak passwords
|
||||
|
||||
<p>
|
||||
<a href="https://weakpass.com/"><b>Weakpass</b></a> - for any kind of bruteforce find wordlists or unleash the power of them all at once!<br>
|
||||
<a href="https://hashes.org/"><b>Hashes.org</b></a> - is a free online hash resolving service incorporating many unparalleled techniques.<br>
|
||||
<a href="https://github.com/danielmiessler/SecLists"><b>SecLists</b></a> - collection of multiple types of lists used during security assessments, collected in one place.<br>
|
||||
<a href="https://github.com/berzerk0/Probable-Wordlists"><b>Probable-Wordlists</b></a> - sorted by probability originally created for password generation and testing.<br>
|
||||
<a href="https://wiki.skullsecurity.org/index.php?title=Passwords"><b>skullsecurity passwords</b></a> - password dictionaries and leaked passwords repository.<br>
|
||||
<a href="https://bezpieka.org/polski-slownik-premium-polish-wordlist"><b>Polish PREMIUM Dictionary</b></a> - official dictionary created by the team on the forum bezpieka.org.<b>*</b> <sup><a href="https://sourceforge.net/projects/kali-linux/files/Wordlist/">1</sup><br>
|
||||
<a href="https://github.com/insidetrust/statistically-likely-usernames"><b>statistically-likely-usernames</b></a> - wordlists for creating statistically likely username lists.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Bounty platforms
|
||||
|
||||
<p>
|
||||
<a href="https://www.yeswehack.com/"><b>YesWeHack</b></a> - bug bounty platform with infosec jobs.<br>
|
||||
<a href="https://www.openbugbounty.org/"><b>Openbugbounty</b></a> - allows any security researcher reporting a vulnerability on any website.<br>
|
||||
<a href="https://www.hackerone.com/"><b>hackerone</b></a> - global hacker community to surface the most relevant security issues.<br>
|
||||
<a href="https://www.bugcrowd.com/"><b>bugcrowd</b></a> - crowdsourced cybersecurity for the enterprise.<br>
|
||||
<a href="https://crowdshield.com/"><b>Crowdshield</b></a> - crowdsourced security & bug bounty management.<br>
|
||||
<a href="https://www.synack.com/"><b>Synack</b></a> - crowdsourced security & bug bounty programs, crowd security intelligence platform, and more.<br>
|
||||
<a href="https://hacktrophy.com/en/"><b>Hacktrophy</b></a> - bug bounty platform.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Web Training Apps (local installation)
|
||||
|
||||
<p>
|
||||
<a href="https://www.owasp.org/index.php/OWASP_Vulnerable_Web_Applications_Directory_Project"><b>OWASP-VWAD</b></a> - comprehensive and well maintained registry of all known vulnerable web applications.<br>
|
||||
<a href="http://www.dvwa.co.uk/"><b>DVWA</b></a> - PHP/MySQL web application that is damn vulnerable.<br>
|
||||
<a href="https://metasploit.help.rapid7.com/docs/metasploitable-2"><b>metasploitable2</b></a> - vulnerable web application amongst security researchers.<br>
|
||||
<a href="https://github.com/rapid7/metasploitable3"><b>metasploitable3</b></a> - is a VM that is built from the ground up with a large amount of security vulnerabilities.<br>
|
||||
<a href="https://github.com/stamparm/DSVW"><b>DSVW</b></a> - is a deliberately vulnerable web application written in under 100 lines of code.<br>
|
||||
<a href="https://sourceforge.net/projects/mutillidae/"><b>OWASP Mutillidae II</b></a> - free, open source, deliberately vulnerable web-application.<br>
|
||||
<a href="https://www.owasp.org/index.php/OWASP_Juice_Shop_Project"><b>OWASP Juice Shop Project</b></a> - the most bug-free vulnerable application in existence.<br>
|
||||
<a href="https://www.owasp.org/index.php/Projects/OWASP_Node_js_Goat_Project"><b>OWASP Node js Goat Project</b></a> - OWASP Top 10 security risks apply to web apps developed using Node.js.<br>
|
||||
<a href="https://github.com/iteratec/juicy-ctf"><b>juicy-ctf</b></a> - run Capture the Flags and Security Trainings with OWASP Juice Shop.<br>
|
||||
<a href="https://github.com/OWASP/SecurityShepherd"><b>SecurityShepherd</b></a> - web and mobile application security training platform.<br>
|
||||
<a href="https://github.com/opendns/Security_Ninjas_AppSec_Training"><b>Security Ninjas</b></a> - open source application security training program.<br>
|
||||
<a href="https://github.com/rapid7/hackazon"><b>hackazon</b></a> - a modern vulnerable web app.<br>
|
||||
<a href="https://github.com/appsecco/dvna"><b>dvna</b></a> - damn vulnerable NodeJS application.<br>
|
||||
<a href="https://github.com/DefectDojo/django-DefectDojo"><b>django-DefectDojo</b></a> - is an open-source application vulnerability correlation and security orchestration tool.<br>
|
||||
<a href="https://google-gruyere.appspot.com/"><b>Google Gruyere</b></a> - web application exploits and defenses.<br>
|
||||
<a href="https://github.com/amolnaik4/bodhi"><b>Bodhi</b></a> - is a playground focused on learning the exploitation of client-side web vulnerabilities.<br>
|
||||
<a href="https://websploit.h4cker.org/"><b>Websploit</b></a> - single vm lab with the purpose of combining several vulnerable appliations in one environment.<br>
|
||||
<a href="https://github.com/vulhub/vulhub"><b>vulhub</b></a> - pre-built Vulnerable Environments based on docker-compose.<br>
|
||||
<a href="https://rhinosecuritylabs.com/aws/introducing-cloudgoat-2/"><b>CloudGoat 2</b></a> - the new & improved "Vulnerable by Design"
|
||||
AWS deployment tool.<br>
|
||||
<a href="https://github.com/globocom/secDevLabs"><b>secDevLabs</b></a> - is a laboratory for learning secure web development in a practical manner.<br>
|
||||
<a href="https://github.com/incredibleindishell/CORS-vulnerable-Lab"><b>CORS-vulnerable-Lab</b></a> - sample vulnerable code and its exploit code.<br>
|
||||
<a href="https://github.com/moloch--/RootTheBox"><b>RootTheBox</b></a> - a Game of Hackers (CTF Scoreboard & Game Manager).<br>
|
||||
<a href="https://application.security/"><b>KONTRA</b></a> - application security training (OWASP Top Web & Api).<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Labs (ethical hacking platforms/trainings/CTFs)
|
||||
|
||||
<p>
|
||||
<a href="https://www.offensive-security.com/"><b>Offensive Security</b></a> - true performance-based penetration testing training for over a decade.<br>
|
||||
<a href="https://www.hackthebox.eu/"><b>Hack The Box</b></a> - online platform allowing you to test your penetration testing skills.<br>
|
||||
<a href="https://www.hacking-lab.com/index.html"><b>Hacking-Lab</b></a> - online ethical hacking, computer network and security challenge platform.<br>
|
||||
<a href="http://pwnable.kr/index.php"><b>pwnable.kr</b></a> - non-commercial wargame site which provides various pwn challenges.<br>
|
||||
<a href="https://pwnable.tw/"><b>Pwnable.tw</b></a> - is a wargame site for hackers to test and expand their binary exploiting skills.<br>
|
||||
<a href="https://picoctf.com/"><b>picoCTF</b></a> - is a free computer security game targeted at middle and high school students.<br>
|
||||
<a href="https://ctflearn.com/"><b>CTFlearn</b></a> - is an online platform built to help ethical hackers learn and practice their cybersecurity knowledge.<br>
|
||||
<a href="https://ctftime.org/"><b>ctftime</b></a> - CTF archive and a place, where you can get some another CTF-related info.<br>
|
||||
<a href="https://silesiasecuritylab.com/"><b>Silesia Security Lab</b></a> - high quality security testing services.<br>
|
||||
<a href="https://practicalpentestlabs.com/"><b>Practical Pentest Labs</b></a> - pentest lab, take your Hacking skills to the next level.<br>
|
||||
<a href="https://www.root-me.org/?lang=en"><b>Root Me</b></a> - the fast, easy, and affordable way to train your hacking skills.<br>
|
||||
<a href="https://rozwal.to/login"><b>rozwal.to</b></a> - a great platform to train your pentesting skills.<br>
|
||||
<a href="https://tryhackme.com/"><b>TryHackMe</b></a> - learning Cyber Security made easy.<br>
|
||||
<a href="https://hackxor.net/"><b>hackxor</b></a> - is a realistic web application hacking game, designed to help players of all abilities develop their skills.<br>
|
||||
<a href="http://hack-yourself-first.com/"><b>Hack Yourself First</b></a> - it's full of nasty app sec holes.<br>
|
||||
<a href="http://overthewire.org/wargames/"><b>OverTheWire</b></a> - can help you to learn and practice security concepts in the form of fun-filled games.<br>
|
||||
<a href="https://labs.wizard-security.net/"><b>Wizard Labs</b></a> - is an online Penetration Testing Lab.<br>
|
||||
<a href="https://pentesterlab.com/"><b>PentesterLab</b></a> - provides vulnerable systems that can be used to test and understand vulnerabilities.<br>
|
||||
<a href="https://ringzer0ctf.com/"><b>RingZer0</b></a> - tons of challenges designed to test and improve your hacking skills.<br>
|
||||
<a href="http://www.try2hack.nl/"><b>try2hack</b></a> - several security-oriented challenges for your entertainment.<br>
|
||||
<a href="https://www.ubeeri.com/preconfig-labs"><b>Ubeeri</b></a> - preconfigured lab environments.<br>
|
||||
<a href="https://lab.pentestit.ru/"><b>Pentestit</b></a> - emulate IT infrastructures of real companies for legal pen testing and improving pentest skills.<br>
|
||||
<a href="https://microcorruption.com/login"><b>Microcorruption</b></a> - reversal challenges done in the web interface.<br>
|
||||
<a href="https://crackmes.one/"><b>Crackmes</b></a> - download crackmes to help improve your reverse engineering skills.<br>
|
||||
<a href="https://domgo.at/cxss/intro"><b>DomGoat</b></a> - DOM XSS security learning and practicing platform.<br>
|
||||
<a href="https://chall.stypr.com"><b>Stereotyped Challenges</b></a> - upgrade your web hacking techniques today!<br>
|
||||
<a href="https://www.vulnhub.com/"><b>Vulnhub</b></a> - allows anyone to gain practical 'hands-on' experience in digital security.<br>
|
||||
<a href="https://w3challs.com/"><b>W3Challs</b></a> - is a penetration testing training platform, which offers various computer challenges.<br>
|
||||
<a href="https://ringzer0ctf.com/challenges"><b>RingZer0 CTF</b></a> - offers you tons of challenges designed to test and improve your hacking skills.<br>
|
||||
<a href="https://hack.me/"><b>Hack.me</b></a> - a platform where you can build, host and share vulnerable web apps for educational purposes.<br>
|
||||
<a href="https://www.hackthis.co.uk/levels/"><b>HackThis!</b></a> - discover how hacks, dumps and defacements are performed and secure your website.<br>
|
||||
<a href="https://www.enigmagroup.org/#"><b>Enigma Group WebApp Training</b></a> - these challenges cover the exploits listed in the OWASP Top 10 Project.<br>
|
||||
<a href="https://challenges.re/"><b>Reverse Engineering Challenges</b></a> - challenges, exercises, problems and tasks - by level, by type, and more.<br>
|
||||
<a href="https://0x00sec.org/"><b>0x00sec</b></a> - the home of the Hacker - Malware, Reverse Engineering, and Computer Science.<br>
|
||||
<a href="https://www.wechall.net/challs"><b>We Chall</b></a> - there are exist a lots of different challenge types.<br>
|
||||
<a href="https://www.hackergateway.com/"><b>Hacker Gateway</b></a> - is the go-to place for hackers who want to test their skills.<br>
|
||||
<a href="https://www.hacker101.com/"><b>Hacker101</b></a> - is a free class for web security.<br>
|
||||
<a href="https://contained.af/"><b>contained.af</b></a> - a stupid game for learning about containers, capabilities, and syscalls.<br>
|
||||
<a href="http://flaws.cloud/"><b>flAWS challenge!</b></a> - a series of levels you'll learn about common mistakes and gotchas when using AWS.<br>
|
||||
<a href="https://cybersecurity.wtf"><b>CyberSec WTF</b></a> - provides web hacking challenges derived from bounty write-ups.<br>
|
||||
<a href="https://ctfchallenge.co.uk/login"><b>CTF Challenge</b></a> - CTF Web App challenges.<br>
|
||||
<a href="https://capturetheflag.withgoogle.com"><b>gCTF</b></a> - most of the challenges used in the Google CTF 2017.<br>
|
||||
<a href="https://www.hackthissite.org/pages/index/index.php"><b>Hack This Site</b></a> - is a free, safe and legal training ground for hackers.<br>
|
||||
<a href="https://attackdefense.com"><b>Attack & Defense</b></a> - is a browser-based cloud labs.<br>
|
||||
<a href="https://cryptohack.org/"><b>Cryptohack</b></a> - a fun platform for learning modern cryptography.<br>
|
||||
<a href="https://cryptopals.com/"><b>Cryptopals</b></a> - the cryptopals crypto challenges.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: CTF platforms
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/facebook/fbctf"><b>fbctf</b></a> - platform to host Capture the Flag competitions.<br>
|
||||
<a href="https://github.com/google/ctfscoreboard"><b>ctfscoreboard</b></a> - scoreboard for Capture The Flag competitions.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Other resources
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/bugcrowd/bugcrowd_university"><b>Bugcrowd University</b></a> - open source education content for the researcher community.<br>
|
||||
<a href="https://github.com/rewardone/OSCPRepo"><b>OSCPRepo</b></a> - a list of resources and scripts that I have been gathering in preparation for the OSCP.<br>
|
||||
<a href="https://medium.com/@cxosmo/owasp-top-10-real-world-examples-part-1-a540c4ea2df5"><b>OWASP Top 10: Real-World Examples</b></a> - test your web apps with real-world examples (two-part series).<br>
|
||||
<a href="http://phrack.org/index.html"><b>phrack.org</b></a> - an awesome collection of articles from several respected hackers and other thinkers.<br>
|
||||
<a href="https://github.com/Gr1mmie/Practical-Ethical-Hacking-Resources"><b>Practical-Ethical-Hacking-Resources</b></a> - compilation of resources from TCM's Udemy Course.<br>
|
||||
</p>
|
||||
|
||||
#### Your daily knowledge and news [<sup>[TOC]</sup>](#anger-table-of-contents)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,97 @@
|
||||
#### Shell Tricks [<sup>[TOC]</sup>](#anger-table-of-contents)
|
||||
|
||||
When you get a shell, it is generally not very clean, but after following these steps, you will have a fairly clean and comfortable shell to work with.
|
||||
|
||||
1) `script /dev/null -c bash`
|
||||
2) Ctrl-Z (to send it to background)
|
||||
3) `stty raw -echo; fg` (returns the shell to foreground)
|
||||
4) `reset` (to reset terminal)
|
||||
5) `xterm` (when asked for terminal type)
|
||||
6) `export TERM=xterm; export SHELL=bash`
|
||||
|
||||
#### Shell functions [<sup>[TOC]</sup>](#anger-table-of-contents)
|
||||
|
||||
##### Table of Contents
|
||||
|
||||
- [Domain resolve](#domain-resolve)
|
||||
- [Get ASN](#get-asn)
|
||||
|
||||
###### Domain resolve
|
||||
|
||||
```bash
|
||||
# Dependencies:
|
||||
# - curl
|
||||
# - jq
|
||||
|
||||
function DomainResolve() {
|
||||
|
||||
local _host="$1"
|
||||
|
||||
local _curl_base="curl --request GET"
|
||||
local _timeout="15"
|
||||
|
||||
_host_ip=$($_curl_base -ks -m "$_timeout" "https://dns.google.com/resolve?name=${_host}&type=A" | \
|
||||
jq '.Answer[0].data' | tr -d "\"" 2>/dev/null)
|
||||
|
||||
if [[ -z "$_host_ip" ]] || [[ "$_host_ip" == "null" ]] ; then
|
||||
|
||||
echo -en "Unsuccessful domain name resolution.\\n"
|
||||
|
||||
else
|
||||
|
||||
echo -en "$_host > $_host_ip\\n"
|
||||
|
||||
fi
|
||||
|
||||
}
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
shell> DomainResolve nmap.org
|
||||
nmap.org > 45.33.49.119
|
||||
|
||||
shell> DomainResolve nmap.org
|
||||
Unsuccessful domain name resolution.
|
||||
```
|
||||
|
||||
###### Get ASN
|
||||
|
||||
```bash
|
||||
# Dependencies:
|
||||
# - curl
|
||||
|
||||
function GetASN() {
|
||||
|
||||
local _ip="$1"
|
||||
|
||||
local _curl_base="curl --request GET"
|
||||
local _timeout="15"
|
||||
|
||||
_asn=$($_curl_base -ks -m "$_timeout" "http://ip-api.com/line/${_ip}?fields=as")
|
||||
|
||||
_state=$(echo $?)
|
||||
|
||||
if [[ -z "$_ip" ]] || [[ "$_ip" == "null" ]] || [[ "$_state" -ne 0 ]]; then
|
||||
|
||||
echo -en "Unsuccessful ASN gathering.\\n"
|
||||
|
||||
else
|
||||
|
||||
echo -en "$_ip > $_asn\\n"
|
||||
|
||||
fi
|
||||
|
||||
}
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
shell> GetASN 1.1.1.1
|
||||
1.1.1.1 > AS13335 Cloudflare, Inc.
|
||||
|
||||
shell> GetASN 0.0.0.0
|
||||
Unsuccessful ASN gathering.
|
||||
```
|
||||
@@ -0,0 +1,269 @@
|
||||
#### Web Tools [<sup>[TOC]</sup>](#anger-table-of-contents)
|
||||
|
||||
##### :black_small_square: Browsers
|
||||
|
||||
<p>
|
||||
<a href="https://www.ssllabs.com/ssltest/viewMyClient.html"><b>SSL/TLS Capabilities of Your Browser</b></a> - test your browser's SSL implementation.<br>
|
||||
<a href="https://caniuse.com/"><b>Can I use</b></a> - provides up-to-date browser support tables for support of front-end web technologies.<br>
|
||||
<a href="https://panopticlick.eff.org/"><b>Panopticlick 3.0</b></a> - is your browser safe against tracking?<br>
|
||||
<a href="https://privacy.net/analyzer/"><b>Privacy Analyzer</b></a> - see what data is exposed from your browser.<br>
|
||||
<a href="https://browserleaks.com/"><b>Web Browser Security</b></a> - it's all about Web Browser fingerprinting.<br>
|
||||
<a href="https://www.howsmyssl.com/"><b>How's My SSL?</b></a> - help a web server developer learn what real world TLS clients were capable of.<br>
|
||||
<a href="https://suche.org/sslClientInfo"><b>sslClientInfo</b></a> - client test (incl TLSv1.3 information).<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: SSL/Security
|
||||
|
||||
<p>
|
||||
<a href="https://www.ssllabs.com/ssltest/"><b>SSLLabs Server Test</b></a> - performs a deep analysis of the configuration of any SSL web server.<br>
|
||||
<a href="https://dev.ssllabs.com/ssltest/"><b>SSLLabs Server Test (DEV)</b></a> - performs a deep analysis of the configuration of any SSL web server.<br>
|
||||
<a href="https://www.immuniweb.com/ssl/"><b>ImmuniWeb® SSLScan</b></a> - test SSL/TLS (PCI DSS, HIPAA and NIST).<br>
|
||||
<a href="https://www.jitbit.com/sslcheck/"><b>SSL Check</b></a> - scan your website for non-secure content.<br>
|
||||
<a href="http://www.ssltools.com"><b>SSL Scanner</b></a> - analyze website security.<br>
|
||||
<a href="https://cryptcheck.fr/"><b>CryptCheck</b></a> - test your TLS server configuration (e.g. ciphers).<br>
|
||||
<a href="https://urlscan.io/"><b>urlscan.io</b></a> - service to scan and analyse websites.<br>
|
||||
<a href="https://report-uri.com/home/tools"><b>Report URI</b></a> - monitoring security policies like CSP and HPKP.<br>
|
||||
<a href="https://csp-evaluator.withgoogle.com/"><b>CSP Evaluator</b></a> - allows developers and security experts to check if a Content Security Policy.<br>
|
||||
<a href="https://uselesscsp.com/"><b>Useless CSP</b></a> - public list about CSP in some big players (might make them care a bit more).<br>
|
||||
<a href="https://whynohttps.com/"><b>Why No HTTPS?</b></a> - top 100 websites by Alexa rank not automatically redirecting insecure requests.<br>
|
||||
<a href="https://ciphersuite.info/"><b>TLS Cipher Suite Search</b></a>- cipher suite search engine.<br>
|
||||
<a href="https://github.com/RaymiiOrg/cipherli.st"><b>cipherli.st</b></a> - strong ciphers for Apache, Nginx, Lighttpd, and more.<b>*</b><br>
|
||||
<a href="https://2ton.com.au/dhtool/"><b>dhtool</b></a> - public Diffie-Hellman parameter service/tool.<br>
|
||||
<a href="https://badssl.com/"><b>badssl.com</b></a> - memorable site for testing clients against bad SSL configs.<br>
|
||||
<a href="https://tlsfun.de/"><b>tlsfun.de</b></a> - registered for various tests regarding the TLS/SSL protocol.<br>
|
||||
<a href="https://sslmate.com/caa/"><b>CAA Record Helper</b></a> - generate a CAA policy.<br>
|
||||
<a href="https://ccadb.org/resources"><b>Common CA Database</b></a> - repository of information about CAs, and their root and intermediate certificates.<br>
|
||||
<a href="https://certstream.calidog.io/"><b>CERTSTREAM</b></a> - real-time certificate transparency log update stream.<br>
|
||||
<a href="https://crt.sh/"><b>crt.sh</b></a> - discovers certificates by continually monitoring all of the publicly known CT.<br>
|
||||
<a href="https://www.hardenize.com/"><b>Hardenize</b></a> - deploy the security standards.<br>
|
||||
<a href="https://cryptcheck.fr/suite/"><b>Cipher suite compatibility</b></a> - test TLS cipher suite compatibility.<br>
|
||||
<a href="https://www.urlvoid.com/"><b>urlvoid</b></a> - this service helps you detect potentially malicious websites.<br>
|
||||
<a href="https://securitytxt.org/"><b>security.txt</b></a> - a proposed standard (generator) which allows websites to define security policies.<br>
|
||||
<a href="https://github.com/mozilla/ssl-config-generator"><b>ssl-config-generator</b></a> - help you follow the Mozilla Server Side TLS configuration guidelines.<br>
|
||||
<a href="https://github.com/mjol/TLScan"><b>TLScan</b></a> - pure python, SSL/TLS protocol and cipher scanner/enumerator.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: HTTP Headers & Web Linters
|
||||
|
||||
<p>
|
||||
<a href="https://securityheaders.com/"><b>Security Headers</b></a> - analyse the HTTP response headers (with rating system to the results).<br>
|
||||
<a href="https://observatory.mozilla.org/"><b>Observatory by Mozilla</b></a> - set of tools to analyze your website.<br>
|
||||
<a href="https://webhint.io/"><b>webhint</b></a> - is a linting tool that will help you with your site's accessibility, speed, security, and more.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: DNS
|
||||
|
||||
<p>
|
||||
<a href="http://viewdns.info/"><b>ViewDNS</b></a> - one source for free DNS related tools and information.<br>
|
||||
<a href="https://dnslookup.org/"><b>DNSLookup</b></a> - is an advanced DNS lookup tool.<br>
|
||||
<a href="https://dnslytics.com/"><b>DNSlytics</b></a> - online DNS investigation tool.<br>
|
||||
<a href="https://dnsspy.io/"><b>DNS Spy</b></a> - monitor, validate and verify your DNS configurations.<br>
|
||||
<a href="https://zonemaster.iis.se/en/"><b>Zonemaster</b></a> - helps you to control how your DNS works.<br>
|
||||
<a href="http://leafdns.com/"><b>Leaf DNS</b></a> - comprehensive DNS tester.<br>
|
||||
<a href="https://findsubdomains.com/"><b>Find subdomains online</b></a> - find subdomains for security assessment penetration test.<br>
|
||||
<a href="https://dnsdumpster.com/"><b>DNSdumpster</b></a> - dns recon & research, find & lookup dns records.<br>
|
||||
<a href="https://dnstable.com/"><b>DNS Table online</b></a> - search for DNS records by domain, IP, CIDR, ISP.<br>
|
||||
<a href="https://intodns.com/"><b>intoDNS</b></a> - DNS and mail server health checker.<br>
|
||||
<a href="http://www.zonecut.net/dns/"><b>DNS Bajaj</b></a> - check the delegation of your domain.<br>
|
||||
<a href="https://www.buddyns.com/delegation-lab/"><b>BuddyDNS Delegation LAB</b></a> - check, trace and visualize delegation of your domain.<br>
|
||||
<a href="https://dnssec-debugger.verisignlabs.com/"><b>dnssec-debugger</b></a> - DS or DNSKEY records validator.<br>
|
||||
<a href="http://ptrarchive.com/"><b>PTRarchive.com</b></a> - this site is responsible for the safekeeping of historical reverse DNS records.<br>
|
||||
<a href="http://xip.io/"><b>xip.io</b></a> - wildcard DNS for everyone.<br>
|
||||
<a href="https://nip.io/"><b>nip.io</b></a> - dead simple wildcard DNS for any IP Address.<br>
|
||||
<a href="https://ceipam.eu/en/dnslookup.php"><b>dnslookup (ceipam)</b></a> - one of the best DNS propagation checker (and not only).<br>
|
||||
<a href="https://whatsmydns.com"><b>What's My DNS</b></a> - DNS propagation checking tool.<br>
|
||||
<a href="https://blog.erbbysam.com/index.php/2019/02/09/dnsgrep/"><b>DNSGrep</b></a> - quickly searching large DNS datasets.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Mail
|
||||
|
||||
<p>
|
||||
<a href="https://luxsci.com/smtp-tls-checker"><b>smtp-tls-checker</b></a> - check an email domain for SMTP TLS support.<br>
|
||||
<a href="https://mxtoolbox.com/SuperTool.aspx"><b>MX Toolbox</b></a> - all of your MX record, DNS, blacklist and SMTP diagnostics in one integrated tool.<br>
|
||||
<a href="https://www.checktls.com/index.html"><b>Secure Email</b></a> - complete email test tools for email technicians.<br>
|
||||
<a href="http://www.blacklistalert.org/"><b>blacklistalert</b></a> - checks to see if your domain is on a Real Time Spam Blacklist.<br>
|
||||
<a href="http://multirbl.valli.org/"><b>MultiRBL</b></a> - complete IP check for sending Mailservers.<br>
|
||||
<a href="https://dkimvalidator.com/"><b>DKIM SPF & Spam Assassin Validator</b></a> - checks mail authentication and scores messages with Spam Assassin.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Encoders/Decoders and Regex testing
|
||||
|
||||
<p>
|
||||
<a href="https://www.url-encode-decode.com/"><b>URL Encode/Decode</b></a> - tool from above to either encode or decode a string of text.<br>
|
||||
<a href="https://uncoder.io/"><b>Uncoder</b></a> - the online translator for search queries on log data.<br>
|
||||
<a href="https://regex101.com/"><b>Regex101</b></a> - online regex tester and debugger: PHP, PCRE, Python, Golang and JavaScript.<br>
|
||||
<a href="https://regexr.com/"><b>RegExr</b></a> - online tool to learn, build, & test Regular Expressions (RegEx / RegExp).<br>
|
||||
<a href="https://www.regextester.com/"><b>RegEx Testing</b></a> - online regex testing tool.<br>
|
||||
<a href="https://www.regexpal.com/"><b>RegEx Pal</b></a> - online regex testing tool + other tools.<br>
|
||||
<a href="https://gchq.github.io/CyberChef/"><b>The Cyber Swiss Army Knife</b></a> - a web app for encryption, encoding, compression and data analysis.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Net-tools
|
||||
|
||||
<p>
|
||||
<a href="https://toolbar.netcraft.com/site_report"><b>Netcraft</b></a> - detailed report about the site, helping you to make informed choices about their integrity.<b>*</b><br>
|
||||
<a href="https://atlas.ripe.net/"><b>RIPE NCC Atlas</b></a> - a global, open, distributed Internet measurement platform.<br>
|
||||
<a href="https://www.robtex.com/"><b>Robtex</b></a> - uses various sources to gather public information about IP numbers, domain names, host names, etc.<br>
|
||||
<a href="https://securitytrails.com/"><b>Security Trails</b></a> - APIs for Security Companies, Researchers and Teams.<br>
|
||||
<a href="https://tools.keycdn.com/curl"><b>Online Curl</b></a> - curl test, analyze HTTP Response Headers.<br>
|
||||
<a href="https://extendsclass.com/"><b>Online Tools for Developers</b></a> - HTTP API tools, testers, encoders, converters, formatters, and other tools.<br>
|
||||
<a href="https://ping.eu/"><b>Ping.eu</b></a> - online Ping, Traceroute, DNS lookup, WHOIS and others.<br>
|
||||
<a href="https://network-tools.com/"><b>Network-Tools</b></a> - network tools for webmasters, IT technicians & geeks.<br>
|
||||
<a href="https://bgpview.io/"><b>BGPview</b></a> - search for any ASN, IP, Prefix or Resource name.<br>
|
||||
<a href="https://isbgpsafeyet.com/"><b>Is BGP safe yet?</b></a> - check BGP (RPKI) security of ISPs and other major Internet players.<br>
|
||||
<a href="https://riseup.net/"><b>Riseup</b></a> - provides online communication tools for people and groups working on liberatory social change.<br>
|
||||
<a href="https://www.virustotal.com/gui/home/upload"><b>VirusTotal</b></a> - analyze suspicious files and URLs to detect types of malware.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Privacy
|
||||
|
||||
<p>
|
||||
<a href="https://www.privacyguides.org/"><b>privacyguides.org</b></a> - provides knowledge and tools to protect your privacy against global mass surveillance.<br>
|
||||
<a href="https://dnsprivacy.org/wiki/display/DP/DNS+Privacy+Test+Servers"><b>DNS Privacy Test Servers</b></a> - DNS privacy recursive servers list (with a 'no logging' policy).<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Code parsers/playgrounds
|
||||
|
||||
<p>
|
||||
<a href="https://www.shellcheck.net/"><b>ShellCheck</b></a> - finds bugs in your shell scripts.<br>
|
||||
<a href="https://explainshell.com/"><b>explainshell</b></a> - get interactive help texts for shell commands.<br>
|
||||
<a href="https://jsbin.com/?html,output"><b>jsbin</b></a> - live pastebin for HTML, CSS & JavaScript, and more.<br>
|
||||
<a href="https://codesandbox.io/"><b>CodeSandbox</b></a> - online code editor for web application development.<br>
|
||||
<a href="http://sandbox.onlinephpfunctions.com/"><b>PHP Sandbox</b></a> - test your PHP code with this code tester.<br>
|
||||
<a href="https://www.repl.it/"><b>Repl.it</b></a> - an instant IDE to learn, build, collaborate, and host all in one place.<br>
|
||||
<a href="http://www.vclfiddle.net/"><b>vclFiddle</b></a> - is an online tool for experimenting with the Varnish Cache VCL.<br>
|
||||
<a href="https://github.com/hadolint/hadolint"><b>Haskell Dockerfile Linter</b></a> - a smarter Dockerfile linter that helps you build best practice Docker images.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Performance
|
||||
|
||||
<p>
|
||||
<a href="https://gtmetrix.com/"><b>GTmetrix</b></a> - analyze your site’s speed and make it faster.<br>
|
||||
<a href="https://performance.sucuri.net/"><b>Sucuri loadtimetester</b></a> - test here the
|
||||
performance of any of your sites from across the globe.<br>
|
||||
<a href="https://tools.pingdom.com/"><b>Pingdom Tools</b></a> - analyze your site’s speed around the world.<br>
|
||||
<a href="https://pingme.io/"><b>PingMe.io</b></a> - run website latency tests across multiple geographic regions.<br>
|
||||
<a href="https://developers.google.com/speed/pagespeed/insights/"><b>PageSpeed Insights</b></a> - analyze your site’s speed and make it faster.<br>
|
||||
<a href="https://web.dev/"><b>web.dev</b></a> - helps developers like you learn and apply the web's modern capabilities to your own sites and apps.<br>
|
||||
<a href="https://github.com/GoogleChrome/lighthouse"><b>Lighthouse</b></a> - automated auditing, performance metrics, and best practices for the web.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Mass scanners (search engines)
|
||||
|
||||
<p>
|
||||
<a href="https://censys.io/"><b>Censys</b></a> - platform that helps information security practitioners discover, monitor, and analyze devices.<br>
|
||||
<a href="https://www.shodan.io/"><b>Shodan</b></a> - the world's first search engine for Internet-connected devices.<br>
|
||||
<a href="https://2000.shodan.io/#/"><b>Shodan 2000</b></a> - this tool looks for randomly generated data from Shodan.<br>
|
||||
<a href="https://viz.greynoise.io/table"><b>GreyNoise</b></a> - mass scanner such as Shodan and Censys.<br>
|
||||
<a href="https://www.zoomeye.org/"><b>ZoomEye</b></a> - search engine for cyberspace that lets the user find specific network components.<br>
|
||||
<a href="https://netograph.io/"><b>netograph</b></a> - tools to monitor and understand deep structure of the web.<br>
|
||||
<a href="https://fofa.so/"><b>FOFA</b></a> - is a cyberspace search engine.<br>
|
||||
<a href="https://www.onyphe.io/"><b>onyphe</b></a> - is a search engine for open-source and cyber threat intelligence data collected.<br>
|
||||
<a href="https://intelx.io/"><b>IntelligenceX</b></a> - is a search engine and data archive.<br>
|
||||
<a href="https://app.binaryedge.io/"><b>binaryedge</b></a> - it scan the entire internet space and create real-time threat intelligence streams and reports.<br>
|
||||
<a href="https://spyse.com/"><b>Spyse</b></a> - Internet assets registry: networks, threats, web objects, etc.<br>
|
||||
<a href="https://wigle.net/"><b>wigle</b></a> - is a submission-based catalog of wireless networks. All the networks. Found by Everyone.<br>
|
||||
<a href="https://publicwww.com/"><b>PublicWWW</b></a> - find any alphanumeric snippet, signature or keyword in the web pages HTML, JS and CSS code.<br>
|
||||
<a href="https://inteltechniques.com/index.html"><b>IntelTechniques</b></a> - this repository contains hundreds of online search utilities.<br>
|
||||
<a href="https://hunter.io/"><b>hunter</b></a> - lets you find email addresses in seconds and connect with the people that matter for your business.<br>
|
||||
<a href="https://ghostproject.fr/"><b>GhostProject?</b></a> - search by full email address or username.<br>
|
||||
<a href="https://www.databreaches.live/"><b>databreaches</b></a> - was my email affected by data breach?<br>
|
||||
<a href="https://weleakinfo.com"><b>We Leak Info</b></a> - world's fastest and largest data breach search engine.<br>
|
||||
<a href="https://pulsedive.com/"><b>Pulsedive</b></a> - scans of malicious URLs, IPs, and domains, including port scans and web requests.<br>
|
||||
<a href="https://buckets.grayhatwarfare.com/"><b>Buckets by Grayhatwarfar</b></a> - database with public search for Open Amazon S3 Buckets and their contents.<br>
|
||||
<a href="https://vigilante.pw/"><b>Vigilante.pw</b></a> - the breached database directory.<br>
|
||||
<a href="https://builtwith.com/"><b>builtwith</b></a> - find out what websites are built with.<br>
|
||||
<a href="https://nerdydata.com/"><b>NerdyData</b></a> - search the web's source code for technologies, across millions of sites.<br>
|
||||
<a href="http://zorexeye.com/"><b>zorexeye</b></a> - search for sites, images, apps, softwares & more.<br>
|
||||
<a href="https://www.mmnt.net/"><b>Mamont's open FTP Index</b></a> - if a target has an open FTP site with accessible content it will be listed here.<br>
|
||||
<a href="https://osintframework.com/"><b>OSINT Framework</b></a> - focused on gathering information from free tools or resources.<br>
|
||||
<a href="https://www.maltiverse.com/search"><b>maltiverse</b></a> - is a service oriented to cybersecurity analysts.<br>
|
||||
<a href="https://leakedsource.ru/main/"><b>Leaked Source</b></a> - is a collaboration of data found online in the form of a lookup.<br>
|
||||
<a href="https://search.weleakinfo.com/"><b>We Leak Info</b></a> - to help everyday individuals secure their online life, avoiding getting hacked.<br>
|
||||
<a href="https://pipl.com/"><b>pipl</b></a> - is the place to find the person behind the email address, social username or phone number.<br>
|
||||
<a href="https://abuse.ch/"><b>abuse.ch</b></a> - is operated by a random swiss guy fighting malware for non-profit.<br>
|
||||
<a href="http://malc0de.com/database/"><b>malc0de</b></a> - malware search engine.<br>
|
||||
<a href="https://cybercrime-tracker.net/index.php"><b>Cybercrime Tracker</b></a> - monitors and tracks various malware families that are used to perpetrate cyber crimes.<br>
|
||||
<a href="https://github.com/eth0izzle/shhgit/"><b>shhgit</b></a> - find GitHub secrets in real time.<br>
|
||||
<a href="https://searchcode.com/"><b>searchcode</b></a> - helping you find real world examples of functions, API's and libraries.<br>
|
||||
<a href="http://www.insecam.org/"><b>Insecam</b></a> - the world biggest directory of online surveillance security cameras.<br>
|
||||
<a href="http://index-of.es/"><b>index-of</b></a> - contains great stuff like: security, hacking, reverse engineering, cryptography, programming etc.<br>
|
||||
<a href="https://opendata.rapid7.com/"><b>Rapid7 Labs Open Data</b></a> - is a great resources of datasets from Project Sonar.<br>
|
||||
<a href="https://webtechsurvey.com/common-response-headers"><b>Common Response Headers</b></a> - the largest database of HTTP response headers.<br>
|
||||
<a href="https://labs.inquest.net"><b>InQuest Labs</b></a> - InQuest Labs is an open, interactive, and API driven data portal for security researchers.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Generators
|
||||
|
||||
<p>
|
||||
<a href="https://thispersondoesnotexist.com/"><b>thispersondoesnotexist</b></a> - generate fake faces in one click - endless possibilities.<br>
|
||||
<a href="https://generated.photos"><b>AI Generated Photos</b></a> - 100.000 AI generated faces.<br>
|
||||
<a href="https://www.fakenamegenerator.com/"><b>fakenamegenerator</b></a> - your randomly generated identity.<br>
|
||||
<a href="https://tools.intigriti.io/redirector/"><b>Intigriti Redirector</b></a> - open redirect/SSRF payload generator.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Passwords
|
||||
|
||||
<p>
|
||||
<a href="https://haveibeenpwned.com/"><b>have i been pwned?</b></a> - check if you have an account that has been compromised in a data breach.<br>
|
||||
<a href="https://www.dehashed.com/"><b>dehashed</b></a> - is a hacked database search engine.<br>
|
||||
<a href="https://leakedsource.ru/"><b>Leaked Source</b></a> - is a collaboration of data found online in the form of a lookup.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: CVE/Exploits databases
|
||||
|
||||
<p>
|
||||
<a href="https://cve.mitre.org/"><b>CVE Mitre</b></a> - list of publicly known cybersecurity vulnerabilities.<br>
|
||||
<a href="https://www.cvedetails.com/"><b>CVE Details</b></a> - CVE security vulnerability advanced database.<br>
|
||||
<a href="https://www.exploit-db.com/"><b>Exploit DB</b></a> - CVE compliant archive of public exploits and corresponding vulnerable software.<br>
|
||||
<a href="https://0day.today/"><b>0day.today</b></a> - exploits market provides you the possibility to buy/sell zero-day exploits.<br>
|
||||
<a href="https://sploitus.com/"><b>sploitus</b></a> - the exploit and tools database.<br>
|
||||
<a href="https://cxsecurity.com/exploit/"><b>cxsecurity</b></a> - free vulnerability database.<br>
|
||||
<a href="https://www.vulncode-db.com/"><b>Vulncode-DB</b></a> - is a database for vulnerabilities and their corresponding source code if available.<br>
|
||||
<a href="https://cveapi.com/"><b>cveapi</b></a> - free API for CVE data.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Mobile apps scanners
|
||||
|
||||
<p>
|
||||
<a href="https://www.immuniweb.com/mobile/"><b>ImmuniWeb® Mobile App Scanner</b></a> - test security and privacy of mobile apps (iOS & Android).<br>
|
||||
<a href="https://vulnerabilitytest.quixxi.com/"><b>Quixxi</b></a> - free Mobile App Vulnerability Scanner for Android & iOS.<br>
|
||||
<a href="https://www.ostorlab.co/scan/mobile/"><b>Ostorlab</b></a> - analyzes mobile application to identify vulnerabilities and potential weaknesses.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Private Search Engines
|
||||
|
||||
<p>
|
||||
<a href="https://www.startpage.com/"><b>Startpage</b></a> - the world's most private search engine.<br>
|
||||
<a href="https://searx.me/"><b>searX</b></a> - a privacy-respecting, hackable metasearch engine.<br>
|
||||
<a href="https://darksearch.io/"><b>darksearch</b></a> - the 1st real Dark Web search engine.<br>
|
||||
<a href="https://www.qwant.com/"><b>Qwant</b></a> - the search engine that respects your privacy.<br>
|
||||
<a href="https://duckduckgo.com/"><b>DuckDuckGo</b></a> - the search engine that doesn't track you.<br>
|
||||
<a href="https://swisscows.com/"><b>Swisscows</b></a> - privacy safe web search<br>
|
||||
<a href="https://search.disconnect.me/"><b>Disconnect</b></a> - the search engine that anonymizes your searches.<br>
|
||||
<a href="https://metager.org/"><b>MetaGer</b></a> - the search engine that uses anonymous proxy and hidden Tor branches.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Secure Webmail Providers
|
||||
|
||||
<p>
|
||||
<a href="https://countermail.com/"><b>CounterMail</b></a> - online email service, designed to provide maximum security and privacy.<br>
|
||||
<a href="http://mail2tor.com/"><b>Mail2Tor</b></a> - is a Tor Hidden Service that allows anyone to send and receive emails anonymously.<br>
|
||||
<a href="https://tutanota.com/"><b>Tutanota</b></a> - is the world's most secure email service and amazingly easy to use.<br>
|
||||
<a href="https://protonmail.com/"><b>Protonmail</b></a> - is the world's largest secure email service, developed by CERN and MIT scientists.<br>
|
||||
<a href="https://www.startmail.com/en/"><b>Startmail</b></a> - private & encrypted email made easy.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: Crypto
|
||||
|
||||
<p>
|
||||
<a href="https://keybase.io/"><b>Keybase</b></a> - it's open source and powered by public-key cryptography.<br>
|
||||
</p>
|
||||
|
||||
##### :black_small_square: PGP Keyservers
|
||||
|
||||
<p>
|
||||
<a href="https://keyserver.ubuntu.com/"><b>SKS OpenPGP Key server</b></a> - services for the SKS keyservers used by OpenPGP.<br>
|
||||
</p>
|
||||
|
||||
#### Systems/Services [<sup>[TOC]</sup>](#anger-table-of-contents)
|
||||
@@ -0,0 +1,31 @@
|
||||
#!/bin/bash
|
||||
# Refresh sysadmin-toolbox from The Book of Secret Knowledge repo
|
||||
# Run periodically to keep references current
|
||||
|
||||
set -e
|
||||
|
||||
REPO_URL="https://github.com/trimstray/the-book-of-secret-knowledge.git"
|
||||
TEMP_DIR="/tmp/tbsk-refresh-$$"
|
||||
SKILL_DIR="${1:-$HOME/clawd-duke-leto/skills/sysadmin-toolbox}"
|
||||
|
||||
echo "🔄 Refreshing sysadmin-toolbox from upstream..."
|
||||
|
||||
# Clone fresh copy
|
||||
git clone --depth 1 "$REPO_URL" "$TEMP_DIR" 2>/dev/null
|
||||
|
||||
cd "$TEMP_DIR"
|
||||
|
||||
# Extract sections
|
||||
echo "📦 Extracting references..."
|
||||
|
||||
awk '/^#### Shell One-liners/,/^#### Shell Tricks/' README.md > "$SKILL_DIR/references/shell-oneliners.md"
|
||||
awk '/^#### Shell Tricks/,/^#### Shell Functions/' README.md > "$SKILL_DIR/references/shell-tricks.md"
|
||||
awk '/^#### CLI Tools/,/^#### GUI Tools/' README.md > "$SKILL_DIR/references/cli-tools.md"
|
||||
awk '/^#### Web Tools/,/^#### Systems\/Services/' README.md > "$SKILL_DIR/references/web-tools.md"
|
||||
awk '/^#### Hacking\/Penetration Testing/,/^#### Your daily knowledge/' README.md > "$SKILL_DIR/references/security-tools.md"
|
||||
|
||||
# Cleanup
|
||||
rm -rf "$TEMP_DIR"
|
||||
|
||||
echo "✅ sysadmin-toolbox refreshed from upstream"
|
||||
echo " Shell one-liners: $(wc -l < "$SKILL_DIR/references/shell-oneliners.md") lines"
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"version": 1,
|
||||
"skills": {
|
||||
"devops-engineer": {
|
||||
"source": "jeffallan/claude-skills",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/devops-engineer/SKILL.md",
|
||||
"computedHash": "b5fefdd43470cf510937c882440fb0759327adfe4a21b511829e545e5faa988e"
|
||||
},
|
||||
"reverse-proxy": {
|
||||
"source": "bagelhole/devops-security-agent-skills",
|
||||
"sourceType": "github",
|
||||
"skillPath": "infrastructure/networking/reverse-proxy/SKILL.md",
|
||||
"computedHash": "07f33fdda533634da44489c6320d67391f1a8e243dd3e06891700e4ea8af72c0"
|
||||
},
|
||||
"sysadmin-toolbox": {
|
||||
"source": "jdrhyne/agent-skills",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/sysadmin-toolbox/SKILL.md",
|
||||
"computedHash": "5e8851b6deb583491dee43ea36389ddbb4cbefa3a0c39d89382a3e79a6f80de4"
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user