chore: add project agent skills

This commit is contained in:
Shaw Kaz
2026-06-08 10:33:30 +08:00
parent cb967a7594
commit 529f6f719b
20 changed files with 6357 additions and 0 deletions
+146
View File
@@ -0,0 +1,146 @@
---
name: devops-engineer
description: Creates Dockerfiles, configures CI/CD pipelines, writes Kubernetes manifests, and generates Terraform/Pulumi infrastructure templates. Handles deployment automation, GitOps configuration, incident response runbooks, and internal developer platform tooling. Use when setting up CI/CD pipelines, containerizing applications, managing infrastructure as code, deploying to Kubernetes clusters, configuring cloud platforms, automating releases, or responding to production incidents. Invoke for pipelines, Docker, Kubernetes, GitOps, Terraform, GitHub Actions, on-call, or platform engineering.
license: MIT
metadata:
author: https://github.com/Jeffallan
version: "1.1.1"
domain: devops
triggers: DevOps, CI/CD, deployment, Docker, Kubernetes, Terraform, GitHub Actions, infrastructure, platform engineering, incident response, on-call, self-service
role: engineer
scope: implementation
output-format: code
related-skills: terraform-engineer, kubernetes-specialist, sre-engineer, monitoring-expert, security-reviewer
---
# DevOps Engineer
Senior DevOps engineer specializing in CI/CD pipelines, infrastructure as code, and deployment automation.
## Role Definition
You are a senior DevOps engineer with 10+ years of experience. You operate with three perspectives:
- **Build Hat**: Automating build, test, and packaging
- **Deploy Hat**: Orchestrating deployments across environments
- **Ops Hat**: Ensuring reliability, monitoring, and incident response
## When to Use This Skill
- Setting up CI/CD pipelines (GitHub Actions, GitLab CI, Jenkins)
- Containerizing applications (Docker, Docker Compose)
- Kubernetes deployments and configurations
- Infrastructure as code (Terraform, Pulumi)
- Cloud platform configuration (AWS, GCP, Azure)
- Deployment strategies (blue-green, canary, rolling)
- Building internal developer platforms and self-service tools
- Incident response, on-call, and production troubleshooting
- Release automation and artifact management
## Core Workflow
1. **Assess** - Understand application, environments, requirements
2. **Design** - Pipeline structure, deployment strategy
3. **Implement** - IaC, Dockerfiles, CI/CD configs
4. **Validate** - Run `terraform plan`, lint configs, execute unit/integration tests; confirm no destructive changes before proceeding
5. **Deploy** - Roll out with verification; run smoke tests post-deployment
6. **Monitor** - Set up observability, alerts; confirm rollback procedure is ready before going live
## Reference Guide
Load detailed guidance based on context:
| Topic | Reference | Load When |
|-------|-----------|-----------|
| GitHub Actions | `references/github-actions.md` | Setting up CI/CD pipelines, GitHub workflows |
| Docker | `references/docker-patterns.md` | Containerizing applications, writing Dockerfiles |
| Kubernetes | `references/kubernetes.md` | K8s deployments, services, ingress, pods |
| Terraform | `references/terraform-iac.md` | Infrastructure as code, AWS/GCP provisioning |
| Deployment | `references/deployment-strategies.md` | Blue-green, canary, rolling updates, rollback |
| Platform | `references/platform-engineering.md` | Self-service infra, developer portals, golden paths, Backstage |
| Release | `references/release-automation.md` | Artifact management, feature flags, multi-platform CI/CD |
| Incidents | `references/incident-response.md` | Production outages, on-call, MTTR, postmortems, runbooks |
## Constraints
### MUST DO
- Use infrastructure as code (never manual changes)
- Implement health checks and readiness probes
- Store secrets in secret managers (not env files)
- Enable container scanning in CI/CD
- Document rollback procedures
- Use GitOps for Kubernetes (ArgoCD, Flux)
### MUST NOT DO
- Deploy to production without explicit approval
- Store secrets in code or CI/CD variables
- Skip staging environment testing
- Ignore resource limits in containers
- Use `latest` tag in production
- Deploy on Fridays without monitoring
## Output Templates
Provide: CI/CD pipeline config, Dockerfile, K8s/Terraform files, deployment verification, rollback procedure
### Minimal GitHub Actions Example
```yaml
name: CI
on:
push:
branches: [main]
jobs:
build-test-push:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Build image
run: docker build -t myapp:${{ github.sha }} .
- name: Run tests
run: docker run --rm myapp:${{ github.sha }} pytest
- name: Scan image
uses: aquasecurity/trivy-action@master
with:
image-ref: myapp:${{ github.sha }}
- name: Push to registry
run: |
docker tag myapp:${{ github.sha }} ghcr.io/org/myapp:${{ github.sha }}
docker push ghcr.io/org/myapp:${{ github.sha }}
```
### Minimal Dockerfile Example
```dockerfile
FROM python:3.12-slim AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
FROM python:3.12-slim
WORKDIR /app
COPY --from=builder /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packages
COPY . .
USER nonroot
HEALTHCHECK --interval=30s --timeout=5s CMD curl -f http://localhost:8080/health || exit 1
CMD ["python", "main.py"]
```
### Rollback Procedure Example
```bash
# Kubernetes: roll back to previous deployment revision
kubectl rollout undo deployment/myapp -n production
kubectl rollout status deployment/myapp -n production
# Verify rollback succeeded
kubectl get pods -n production -l app=myapp
curl -f https://myapp.example.com/health
```
Always document the rollback command and verification step in the PR or change ticket before deploying.
## Knowledge Reference
GitHub Actions, GitLab CI, Jenkins, CircleCI, Docker, Kubernetes, Helm, ArgoCD, Flux, Terraform, Pulumi, Crossplane, AWS/GCP/Azure, Prometheus, Grafana, PagerDuty, Backstage, LaunchDarkly, Flagger
[Documentation](https://jeffallan.github.io/claude-skills/skills/devops/devops-engineer/)
@@ -0,0 +1,241 @@
# Deployment Strategies
## Strategy Comparison
| Strategy | Use When | Rollback | Risk |
|----------|----------|----------|------|
| **Rolling** | Standard updates, can tolerate mixed versions | Automatic via health checks | Low |
| **Blue-Green** | Zero downtime, instant rollback needed | Switch traffic to old env | Medium |
| **Canary** | Risk mitigation, gradual rollout | Scale down canary | Low |
| **Recreate** | Stateful apps, breaking changes | Redeploy previous version | High |
## Rolling Deployment (Kubernetes)
```yaml
apiVersion: apps/v1
kind: Deployment
spec:
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 25% # Max pods above desired
maxUnavailable: 25% # Max pods unavailable
```
## Blue-Green with Ingress
```yaml
# Blue deployment (current)
apiVersion: apps/v1
kind: Deployment
metadata:
name: app-blue
labels:
version: blue
---
# Green deployment (new)
apiVersion: apps/v1
kind: Deployment
metadata:
name: app-green
labels:
version: green
---
# Service pointing to active version
apiVersion: v1
kind: Service
metadata:
name: app
spec:
selector:
version: blue # Switch to 'green' for cutover
```
## Canary with Istio
```yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: app
spec:
hosts:
- app
http:
- match:
- headers:
canary:
exact: "true"
route:
- destination:
host: app-canary
- route:
- destination:
host: app-stable
weight: 90
- destination:
host: app-canary
weight: 10
```
## Rollback Procedures
### Kubernetes Rollback
```bash
# View rollout history
kubectl rollout history deployment/app
# Rollback to previous
kubectl rollout undo deployment/app
# Rollback to specific revision
kubectl rollout undo deployment/app --to-revision=2
# Check status
kubectl rollout status deployment/app
```
### ArgoCD Rollback
```bash
argocd app rollback app-prod --revision=123
```
### Terraform Rollback
```bash
# Identify previous state
terraform state list
# Import previous configuration
git checkout HEAD~1 -- main.tf
terraform apply
```
## Pre-deployment Checklist
- [ ] Database migrations are backward compatible
- [ ] Feature flags for new functionality
- [ ] Monitoring dashboards updated
- [ ] Alert thresholds reviewed
- [ ] Rollback procedure documented
- [ ] Staging tested and approved
- [ ] Team notified of deployment window
## Post-deployment Verification
```bash
# Check pod status
kubectl get pods -l app=app
# Check logs for errors
kubectl logs -l app=app --tail=100 | grep -i error
# Verify endpoints
curl -f https://app.example.com/health
# Check metrics
# - Error rate < 1%
# - Latency p99 < 500ms
# - No memory/CPU spikes
```
## Deployment Metrics (DORA)
Track four key metrics:
- **Deployment Frequency**: Target 10+/day
- **Lead Time for Changes**: Target <1 hour
- **Change Failure Rate**: Target <5%
- **MTTR**: Target <30 minutes
```yaml
# Prometheus metrics for DORA tracking
- record: deployment:frequency:1d
expr: count_over_time(deployment_completed[1d])
- record: deployment:lead_time:p95
expr: histogram_quantile(0.95,
rate(commit_to_deploy_seconds_bucket[1h]))
- record: deployment:failure_rate
expr: |
sum(rate(deployment_failed[1h]))
/ sum(rate(deployment_total[1h]))
```
## Advanced Canary with Automated Analysis
```yaml
# Flagger: Automated canary with rollback
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
name: api
spec:
provider: istio
targetRef:
apiVersion: apps/v1
kind: Deployment
name: api
progressDeadlineSeconds: 60
service:
port: 8080
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
analysis:
interval: 30s
threshold: 5
maxWeight: 50
stepWeight: 10
metrics:
- name: error-rate
templateRef:
name: error-rate
thresholdRange:
max: 1
- name: latency
templateRef:
name: latency
thresholdRange:
max: 500
webhooks:
- name: acceptance-test
type: pre-rollout
url: http://test-runner/
- name: load-test
url: http://loadtester/
timeout: 5s
metadata:
type: bash
cmd: "hey -z 1m -q 10 http://api-canary:8080/"
```
## Shadow Deployment
```yaml
# Mirror traffic to shadow deployment
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: api
spec:
hosts:
- api
http:
- match:
- headers:
x-test-version:
exact: "v2"
route:
- destination:
host: api
subset: v2
mirror:
host: api
subset: v2-shadow
mirrorPercentage:
value: 100
- route:
- destination:
host: api
subset: v1
```
@@ -0,0 +1,113 @@
# Docker Patterns
## Multi-stage Dockerfile (Node.js)
```dockerfile
# Build stage
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production && npm cache clean --force
COPY . .
RUN npm run build
# Production stage
FROM node:20-alpine AS runner
WORKDIR /app
ENV NODE_ENV=production
RUN addgroup -g 1001 -S nodejs && adduser -S nodejs -u 1001
COPY --from=builder --chown=nodejs:nodejs /app/dist ./dist
COPY --from=builder --chown=nodejs:nodejs /app/node_modules ./node_modules
USER nodejs
EXPOSE 3000
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s \
CMD wget -qO- http://localhost:3000/health || exit 1
CMD ["node", "dist/main.js"]
```
## Multi-stage Dockerfile (Python)
```dockerfile
FROM python:3.12-slim AS builder
WORKDIR /app
RUN pip install --no-cache-dir poetry
COPY pyproject.toml poetry.lock ./
RUN poetry export -f requirements.txt --output requirements.txt
RUN pip wheel --no-cache-dir --wheel-dir /wheels -r requirements.txt
FROM python:3.12-slim AS runner
WORKDIR /app
RUN useradd -m -u 1001 appuser
COPY --from=builder /wheels /wheels
RUN pip install --no-cache-dir /wheels/*
COPY --chown=appuser:appuser . .
USER appuser
EXPOSE 8000
HEALTHCHECK --interval=30s --timeout=3s \
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
```
## Docker Compose (Development)
```yaml
version: '3.8'
services:
app:
build:
context: .
target: builder # Use dev stage
volumes:
- .:/app
- /app/node_modules
ports:
- "3000:3000"
environment:
- DATABASE_URL=postgres://user:pass@db:5432/app
depends_on:
db:
condition: service_healthy
db:
image: postgres:16-alpine
environment:
POSTGRES_USER: user
POSTGRES_PASSWORD: pass
POSTGRES_DB: app
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U user -d app"]
interval: 5s
timeout: 5s
retries: 5
volumes:
postgres_data:
```
## Security Best Practices
| Practice | Implementation |
|----------|----------------|
| Non-root user | `USER nodejs` or `USER 1001` |
| Minimal base image | Use `-alpine` or `-slim` variants |
| No secrets in image | Use runtime env vars or secrets |
| Pin versions | `FROM node:20.10.0-alpine` not `latest` |
| Scan images | `docker scout`, `trivy`, `snyk` |
| Health checks | `HEALTHCHECK` instruction |
| .dockerignore | Exclude `node_modules`, `.git`, `.env` |
## .dockerignore Template
```
node_modules
.git
.env*
*.md
Dockerfile*
docker-compose*
.dockerignore
coverage
.nyc_output
```
@@ -0,0 +1,139 @@
# GitHub Actions Pipelines
## Complete CI/CD Pipeline
```yaml
name: CI/CD Pipeline
on:
push:
branches: [main, develop]
pull_request:
branches: [main]
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
- run: npm ci
- run: npm test
- run: npm run lint
build:
needs: test
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
outputs:
image-tag: ${{ steps.meta.outputs.tags }}
steps:
- uses: actions/checkout@v4
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=sha,prefix=
type=ref,event=branch
- uses: docker/build-push-action@v5
with:
context: .
push: true
tags: ${{ steps.meta.outputs.tags }}
cache-from: type=gha
cache-to: type=gha,mode=max
deploy-staging:
needs: build
if: github.ref == 'refs/heads/develop'
runs-on: ubuntu-latest
environment: staging
steps:
- uses: actions/checkout@v4
- run: |
kubectl set image deployment/app app=${{ needs.build.outputs.image-tag }}
deploy-production:
needs: build
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
environment: production
steps:
- uses: actions/checkout@v4
- run: |
kubectl set image deployment/app app=${{ needs.build.outputs.image-tag }}
```
## Common Workflow Patterns
### Matrix Builds (Multi-version testing)
```yaml
jobs:
test:
strategy:
matrix:
node-version: [18, 20, 22]
os: [ubuntu-latest, macos-latest]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/setup-node@v4
with:
node-version: ${{ matrix.node-version }}
```
### Reusable Workflows
```yaml
# .github/workflows/deploy.yml
on:
workflow_call:
inputs:
environment:
required: true
type: string
secrets:
DEPLOY_KEY:
required: true
jobs:
deploy:
runs-on: ubuntu-latest
environment: ${{ inputs.environment }}
steps:
- run: echo "Deploying to ${{ inputs.environment }}"
```
### Caching Dependencies
```yaml
- uses: actions/cache@v4
with:
path: ~/.npm
key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-node-
```
## Quick Reference
| Action | Purpose |
|--------|---------|
| `actions/checkout@v4` | Clone repository |
| `actions/setup-node@v4` | Install Node.js |
| `docker/build-push-action@v5` | Build and push Docker image |
| `docker/metadata-action@v5` | Generate Docker tags |
| `actions/cache@v4` | Cache dependencies |
@@ -0,0 +1,331 @@
# Incident Response
## Response Metrics
- **MTTD** (Mean Time to Detect): Target < 5 minutes
- **MTTA** (Mean Time to Acknowledge): Target < 5 minutes
- **MTTR** (Mean Time to Resolve): Target < 30 minutes
- **MTBF** (Mean Time Between Failures): Maximize
### Severity Levels
| Level | Impact | Response | Example |
|-------|--------|----------|---------|
| SEV1 | Complete outage | Immediate | Database down, payment failed |
| SEV2 | Major degradation | 15 min | API latency >5s, 50% errors |
| SEV3 | Minor degradation | 1 hour | Non-critical feature broken |
| SEV4 | Low impact | Business hours | UI glitch, logging issues |
## Runbook Template
```markdown
# Runbook: High API Error Rate
## Symptoms
- Alert: `api_error_rate > 0.05`
- Dashboard: https://grafana.example.com/d/api-errors
## Impact
Users cannot complete purchases (~$X per minute)
## Triage
1. Check dashboard for affected endpoints
2. Check recent deployments: `kubectl rollout history deployment/api`
3. Check dependencies: database, redis, external APIs
## Resolution
### Option 1: Rollback
kubectl rollout undo deployment/api -n production
### Option 2: Scale Up
kubectl scale deployment/api --replicas=10 -n production
### Option 3: Fix Config
kubectl set env deployment/api DB_POOL_SIZE=50 -n production
## Verification
- [ ] Error rate <1%
- [ ] P95 latency <500ms
- [ ] Health checks passing
## Communication
- Update status page
- Notify #incidents
- Post if user-facing
```
## Auto-Remediation Script
```python
#!/usr/bin/env python3
import kubernetes, prometheus_api_client
class IncidentRemediator:
def check_high_error_rate(self):
query = 'rate(http_requests_total{status=~"5.."}[5m]) > 0.05'
result = self.prometheus.custom_query(query)
return len(result) > 0
def rollback_deployment(self, namespace, deployment):
body = {'spec': {'rollbackTo': {'revision': 0}}}
self.k8s.patch_namespaced_deployment(deployment, namespace, body)
def remediate(self):
if self.check_high_error_rate():
if self.rollback_deployment('production', 'api'):
time.sleep(120)
if not self.check_high_error_rate():
return # Success
# Escalate if remediation fails
self.create_incident("Auto-remediation failed")
```
## Postmortem Template
```markdown
# Postmortem: API Outage - 2024-01-15
**Date**: January 15, 2024
**Duration**: 45 minutes (14:23 - 15:08 UTC)
**Severity**: SEV1
**Impact**: Complete API outage, ~$25K revenue loss
## Summary
API became unresponsive due to database connection pool exhaustion
from slow query in v2.3.1.
## Timeline (UTC)
- 14:23 - Alert fired
- 14:27 - Incident declared SEV1
- 14:30 - Rollback initiated
- 14:45 - Identified slow query
- 14:50 - Killed queries
- 15:08 - Resolved
## Root Cause
New query missing index on `user_id`, causing full table scans that
exhausted connection pool under load.
## Impact
- 100% API failure for 45 minutes
- 15,000 users affected
- $25K revenue loss
- 200+ support tickets
## Action Items
| Action | Owner | Deadline |
|--------|-------|----------|
| Add index on user_id | DB team | 2024-01-16 |
| Add query perf testing | Platform | 2024-01-22 |
| Increase staging DB size | Infra | 2024-01-30 |
## Lessons Learned
- Performance testing must use production-scale data
- Connection pool exhaustion needs active intervention
- Consider circuit breakers for DB operations
```
## PagerDuty Configuration
```yaml
schedules:
- name: Primary On-Call
time_zone: America/New_York
layers:
- rotation_turn_length_seconds: 604800 # 1 week
users: [PXXXXXX, PXXXXXX, PXXXXXX]
escalation_policies:
- name: Production
rules:
- escalation_delay_in_minutes: 0
targets: [{type: schedule, id: primary}]
- escalation_delay_in_minutes: 15
targets: [{type: schedule, id: secondary}]
- escalation_delay_in_minutes: 30
targets: [{type: user, id: manager}]
```
## Chaos Engineering
```yaml
# chaos-mesh: Pod failure test
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
name: pod-failure-test
spec:
action: pod-failure
mode: one
duration: "30s"
selector:
namespaces: [production]
labelSelectors:
app: api
scheduler:
cron: "@every 2h"
```
```bash
#!/bin/bash
# Game Day: Database failover drill
echo "🎮 Game Day: Database failover"
slack-cli -d incidents "Starting failover drill"
# Simulate failure
kubectl delete pod postgres-0 -n production
# Monitor recovery
start=$(date +%s)
while ! kubectl get pod postgres-1 | grep Running; do
sleep 5
done
duration=$(($(date +%s) - start))
echo "Failover: ${duration}s" >> results.md
curl -f https://api.example.com/health || echo "FAIL"
```
## Evidence Collection & Forensics
```bash
#!/bin/bash
# collect-evidence.sh - Preserve incident evidence
INCIDENT_ID=$1
EVIDENCE_DIR="incidents/${INCIDENT_ID}/evidence"
mkdir -p $EVIDENCE_DIR
# Preserve logs
kubectl logs -l app=api --all-containers --timestamps \
--since=2h > $EVIDENCE_DIR/pod-logs.txt
# Capture current state
kubectl get all -n production -o yaml > $EVIDENCE_DIR/k8s-state.yaml
kubectl describe pods -n production > $EVIDENCE_DIR/pod-details.txt
# Network traces
kubectl exec -n production deploy/api -- \
tcpdump -i any -w /tmp/capture.pcap -G 60 -W 5 &
# Memory/CPU snapshot
kubectl top pods -n production > $EVIDENCE_DIR/resource-usage.txt
# Git commit at time of incident
git log --since="2 hours ago" --oneline > $EVIDENCE_DIR/recent-commits.txt
# Database queries
psql -c "SELECT * FROM pg_stat_activity" > $EVIDENCE_DIR/db-activity.txt
# Create timeline
echo "$(date): Evidence collection completed" >> $EVIDENCE_DIR/timeline.txt
```
## Communication Templates
```markdown
## SEV1 Initial Notification
**INCIDENT ALERT - SEV1**
**Status**: Investigating
**Impact**: Payment API unavailable (100% error rate)
**Started**: 2024-01-15 14:23 UTC
**Affected**: All users (~15K active sessions)
**Lead**: @oncall-engineer
**War Room**: https://zoom.us/incident-123
Updates every 15 minutes or on major change.
---
## SEV1 Resolution Notification
**INCIDENT RESOLVED**
**Summary**: Payment API restored after database connection pool exhaustion
**Duration**: 45 minutes (14:23 - 15:08 UTC)
**Resolution**: Rollback to v2.3.0 + query optimization
**Impact**: 15K users, ~$25K revenue loss
**Next Steps**: Postmortem scheduled for Jan 16 10am
Thanks to @oncall-team for rapid response.
```
## Incident Classification
| Type | Examples | Response Team | Escalation |
|------|----------|---------------|------------|
| **Security** | Breach, data leak, unauthorized access | Security + DevOps | CISO, Legal |
| **Service** | Outage, degradation, errors | DevOps + SRE | Engineering VP |
| **Data** | Corruption, loss, sync issues | DBA + DevOps | CTO |
| **Compliance** | GDPR, SOC2, audit violations | Compliance + Legal | CEO |
| **Third-party** | Provider outage, API failures | DevOps + Product | Account manager |
## Security Incident Specifics
```bash
# Compromise investigation checklist
□ Isolate affected systems
□ Preserve logs and memory dumps
□ Identify attack vector
□ Check for lateral movement
□ Scan for malware/backdoors
□ Review access logs for 30 days
□ Identify data accessed
□ Assess exfiltration risk
□ Check for persistence mechanisms
□ Coordinate with security team
□ Notify legal if PII involved
□ Document chain of custody
```
## Compliance Requirements
```yaml
# Incident notification requirements
gdpr:
notification_deadline: 72h
authority: Data Protection Officer
required_info:
- Nature of breach
- Data categories affected
- Number of individuals
- Consequences
- Remediation measures
sox:
notification_deadline: immediate
authority: Audit Committee
documentation:
- Financial impact
- Control failures
- Remediation plan
pci_dss:
notification_deadline: 24h
authority: Card brands + acquirer
required_info:
- Cardholder data affected
- Incident timeline
- Forensic investigation
```
## Best Practices
- Maintain runbooks for critical services
- Practice with game days monthly
- Automate common remediation
- Keep postmortems blameless
- Track incident metrics
- Test recovery procedures
- Document all incidents
- Improve detection continuously
- Preserve evidence chain properly
- Coordinate communication clearly
- Escalate security incidents immediately
- Understand compliance obligations
- Train team on response procedures
- Review and update playbooks quarterly
@@ -0,0 +1,154 @@
# Kubernetes Manifests
## Complete Deployment Stack
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: app
labels:
app: app
spec:
replicas: 3
selector:
matchLabels:
app: app
template:
metadata:
labels:
app: app
spec:
containers:
- name: app
image: ghcr.io/org/app:latest
ports:
- containerPort: 3000
resources:
requests:
memory: "128Mi"
cpu: "100m"
limits:
memory: "256Mi"
cpu: "500m"
livenessProbe:
httpGet:
path: /health
port: 3000
initialDelaySeconds: 10
periodSeconds: 10
readinessProbe:
httpGet:
path: /ready
port: 3000
initialDelaySeconds: 5
periodSeconds: 5
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: app-secrets
key: database-url
---
apiVersion: v1
kind: Service
metadata:
name: app
spec:
selector:
app: app
ports:
- port: 80
targetPort: 3000
type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: app
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
ingressClassName: nginx
tls:
- hosts: [app.example.com]
secretName: app-tls
rules:
- host: app.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: app
port:
number: 80
```
## ConfigMap and Secrets
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: app-config
data:
LOG_LEVEL: "info"
API_TIMEOUT: "30s"
---
apiVersion: v1
kind: Secret
metadata:
name: app-secrets
type: Opaque
stringData:
database-url: "postgres://user:pass@host:5432/db"
```
## Horizontal Pod Autoscaler
```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: app-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: app
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
```
## Quick Reference
| Resource | Purpose |
|----------|---------|
| Deployment | Manages ReplicaSets, rolling updates |
| Service | Internal load balancing, DNS |
| Ingress | External HTTP/HTTPS routing |
| ConfigMap | Non-sensitive configuration |
| Secret | Sensitive data (base64 encoded) |
| HPA | Auto-scaling based on metrics |
| PVC | Persistent storage claims |
## Common kubectl Commands
```bash
kubectl apply -f deployment.yaml
kubectl get pods -l app=app
kubectl describe pod <pod-name>
kubectl logs -f <pod-name>
kubectl exec -it <pod-name> -- /bin/sh
kubectl rollout status deployment/app
kubectl rollout undo deployment/app
```
@@ -0,0 +1,417 @@
# Platform Engineering
## Platform Principles
- **Self-service first**: Reduce manual work to <10%
- **Golden paths**: Pre-approved, opinionated templates
- **Developer experience**: Measure and optimize productivity
- **Platform as product**: Treat with product mindset
## Self-Service with Crossplane
```yaml
# Composition for self-service database
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
name: postgres-database
spec:
compositeTypeRef:
apiVersion: platform.example.com/v1alpha1
kind: Database
resources:
- name: rds-instance
base:
apiVersion: rds.aws.crossplane.io/v1alpha1
kind: DBInstance
spec:
forProvider:
dbInstanceClass: db.t3.micro
engine: postgres
engineVersion: "15"
masterUsername: admin
allocatedStorage: 20
```
## Terraform Self-Service Module
```hcl
# modules/service/main.tf
variable "service_name" {}
variable "environment" {}
module "k8s_service" {
source = "./k8s-deployment"
name = var.service_name
env = var.environment
}
module "database" {
source = "./postgres"
name = "${var.service_name}-db"
}
module "monitoring" {
source = "./monitoring-stack"
service = var.service_name
}
output "service_url" {
value = module.k8s_service.url
}
```
## Backstage Service Template
```yaml
# templates/microservice/template.yaml
apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
name: microservice-template
title: Microservice Golden Path
spec:
owner: platform-team
type: service
parameters:
- title: Service Info
properties:
name:
type: string
owner:
type: string
ui:field: OwnerPicker
language:
type: string
enum: [go, python, nodejs, java]
steps:
- id: fetch
action: fetch:template
input:
url: ./skeleton
values:
name: ${{ parameters.name }}
- id: publish
action: publish:github
input:
repoUrl: github.com?owner=org&repo=${{ parameters.name }}
- id: register
action: catalog:register
```
## Service Catalog Info
```yaml
# catalog-info.yaml
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
name: payment-service
annotations:
github.com/project-slug: org/payment-service
pagerduty.com/integration-key: abc123
grafana/dashboard-selector: service=payment
spec:
type: service
lifecycle: production
owner: payments-team
system: checkout
dependsOn:
- resource:default/payment-db
- component:default/auth-service
providesApis:
- payment-api
```
## Golden Path Scaffolding
```bash
#!/bin/bash
# create-service.sh - Golden path for new services
SERVICE=$1
LANG=$2
# Create from template
gh repo create "org/$SERVICE" --template "org/template-$LANG"
git clone "git@github.com:org/$SERVICE.git"
cd "$SERVICE"
# Setup CI/CD
cat > .github/workflows/ci.yml <<EOF
name: CI/CD
on: [push]
jobs:
pipeline:
uses: org/workflows/.github/workflows/standard.yml@v1
with:
service_name: $SERVICE
EOF
# Create infrastructure
cat > terraform/main.tf <<EOF
module "service" {
source = "git::https://github.com/org/terraform//service"
name = "$SERVICE"
}
EOF
git add . && git commit -m "Golden path init" && git push
echo "✓ Service created! Merge to main to deploy."
```
## GitOps Repository Structure
```
gitops/
├── apps/
│ ├── production/
│ │ ├── payment-service/
│ │ └── auth-service/
│ └── staging/
│ └── payment-service/
├── infrastructure/
│ ├── clusters/
│ │ ├── prod-us-east/
│ │ └── prod-eu-west/
│ └── base/
│ ├── ingress/
│ └── monitoring/
└── platform/
├── backstage/
├── argocd/
└── vault/
```
## ArgoCD Application
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: payment-service
spec:
project: default
source:
repoURL: https://github.com/org/gitops
path: apps/production/payment-service
destination:
server: https://kubernetes.default.svc
namespace: production
syncPolicy:
automated:
prune: true
selfHeal: true
retry:
limit: 5
backoff:
duration: 5s
maxDuration: 3m
```
## Platform Metrics
```yaml
# prometheus/platform-metrics.yaml
groups:
- name: platform
rules:
# Self-service adoption rate
- record: platform:self_service:rate
expr: |
sum(rate(platform_provision_automated[1h]))
/
sum(rate(platform_provision_total[1h]))
# Provisioning time P95
- record: platform:provision:p95
expr: |
histogram_quantile(0.95,
rate(platform_provision_duration_bucket[5m]))
# Golden path adoption
- record: platform:golden_path:adoption
expr: |
count(service{template="golden-path"})
/ count(service)
```
## Custom Backstage Plugin
```typescript
// plugins/platform-stats/PlatformMetrics.tsx
import React from 'react';
import { InfoCard, Progress } from '@backstage/core-components';
export const PlatformMetrics = () => {
const metrics = {
selfServiceRate: 92,
avgProvisionTime: '3.5min',
uptime: '99.95%',
satisfaction: 4.6
};
return (
<InfoCard title="Platform Health">
<Progress value={metrics.selfServiceRate} label="Self-Service" />
<p>Provision Time: {metrics.avgProvisionTime}</p>
<p>Uptime: {metrics.uptime}</p>
<p>Satisfaction: {metrics.satisfaction}/5</p>
</InfoCard>
);
};
```
## Cost Allocation
```yaml
# kubecost/allocation.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: cost-allocation
data:
allocation.json: |
{
"defaultLabels": {
"team": "team",
"service": "app",
"environment": "env"
},
"shareNamespaces": ["kube-system"],
"shareCost": "weighted"
}
```
## Platform APIs
```python
# Platform API for self-service provisioning
from fastapi import FastAPI, Depends
from pydantic import BaseModel
app = FastAPI()
class ServiceRequest(BaseModel):
name: str
environment: str
language: str
database: bool = False
@app.post("/api/v1/services")
async def create_service(request: ServiceRequest):
# Validate and enqueue
task = platform.provision_service(
name=request.name,
env=request.environment,
template=f"golden-path-{request.language}"
)
return {"task_id": task.id, "status": "provisioning"}
@app.get("/api/v1/services/{name}/status")
async def service_status(name: str):
return {
"status": "running",
"url": f"https://{name}.example.com",
"health": "healthy",
"cost_mtd": "$142.50"
}
```
## Multi-Tenant Architecture
```yaml
# Policy: Resource quotas per tenant
apiVersion: v1
kind: ResourceQuota
metadata:
name: team-quota
namespace: team-payments
spec:
hard:
requests.cpu: "20"
requests.memory: 40Gi
persistentvolumeclaims: "10"
services.loadbalancers: "2"
---
# RBAC: Namespace admin
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: team-admin
namespace: team-payments
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: namespace-admin
subjects:
- kind: Group
name: team-payments
```
## Adoption Strategy
```yaml
# Platform metrics tracking
apiVersion: v1
kind: ConfigMap
metadata:
name: platform-goals
data:
goals.yaml: |
q1_2024:
self_service_rate: 90%
avg_provision_time: 5min
developer_satisfaction: 4.5/5
golden_path_adoption: 80%
tracking:
weekly_provisioning: true
team_feedback: true
support_tickets: true
training_completion: true
```
## CLI Tool Example
```bash
#!/bin/bash
# platform-cli - Self-service CLI
platform() {
case $1 in
create)
curl -X POST $PLATFORM_API/services \
-d "{\"name\":\"$2\",\"env\":\"$3\",\"language\":\"$4\"}"
;;
status)
curl $PLATFORM_API/services/$2/status | jq
;;
logs)
kubectl logs -l app=$2 -n ${3:-staging} --tail=100
;;
cost)
curl $PLATFORM_API/services/$2/cost?period=mtd
;;
esac
}
```
## Best Practices
- Design for self-service from day one
- Make golden paths the easiest option
- Measure developer satisfaction continuously
- Automate platform operations
- Provide excellent documentation
- Build APIs, not just tools
- Enable safe experimentation
- Maintain backward compatibility
- Treat platform as a product
- Gather and act on feedback
- Track adoption metrics weekly
- Run platform as a product team
- Invest in developer evangelism
- Maintain SLOs for platform uptime
- Provide fast, helpful support
@@ -0,0 +1,527 @@
# Release Automation
## Artifact Management
### Container Registry Lifecycle
```json
{
"rules": [
{
"rulePriority": 1,
"description": "Keep last 10 prod images",
"selection": {
"tagStatus": "tagged",
"tagPrefixList": ["prod-"],
"countType": "imageCountMoreThan",
"countNumber": 10
},
"action": {"type": "expire"}
},
{
"rulePriority": 2,
"description": "Remove untagged after 7 days",
"selection": {
"tagStatus": "untagged",
"countType": "sinceImagePushed",
"countUnit": "days",
"countNumber": 7
},
"action": {"type": "expire"}
}
]
}
```
### Artifact Promotion
```yaml
# .github/workflows/promote.yml
name: Artifact Promotion
on:
workflow_dispatch:
inputs:
image_tag:
required: true
target_env:
type: choice
options: [staging, production]
jobs:
promote:
runs-on: ubuntu-latest
steps:
- name: Re-tag for environment
run: |
docker pull $REGISTRY/$IMAGE:${{ inputs.image_tag }}
docker tag $REGISTRY/$IMAGE:${{ inputs.image_tag }} \
$REGISTRY/$IMAGE:${{ inputs.target_env }}-latest
docker push $REGISTRY/$IMAGE:${{ inputs.target_env }}-latest
- name: Sign artifact
uses: sigstore/cosign-installer@v3
- run: cosign sign $REGISTRY/$IMAGE:${{ inputs.target_env }}-latest
- name: Update GitOps
run: |
cd gitops/apps/${{ inputs.target_env }}
yq e '.image.tag = "${{ inputs.image_tag }}"' -i values.yaml
git commit -am "Promote to ${{ inputs.target_env }}"
git push
```
## Feature Flags
### LaunchDarkly Integration
```python
import launchdarkly
ld = launchdarkly.get()
def should_enable(user_id, feature_key):
user = {"key": user_id, "custom": {"groups": get_groups(user_id)}}
return ld.variation(feature_key, user, False)
# Usage
if should_enable(user.id, "new-payment-flow"):
return new_payment_service.process(payment)
else:
return legacy_payment_service.process(payment)
```
### Flagger Progressive Delivery
```yaml
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
name: payment-service
spec:
targetRef:
kind: Deployment
name: payment-service
service:
port: 8080
analysis:
interval: 1m
threshold: 5
maxWeight: 50
stepWeight: 10
metrics:
- name: request-success-rate
thresholdRange:
min: 99
- name: request-duration
thresholdRange:
max: 500
webhooks:
- name: load-test
url: http://flagger-loadtester/
metadata:
cmd: "hey -z 1m -q 10 http://payment-canary/"
```
## Multi-Platform CI/CD
### GitLab CI
```yaml
stages: [test, build, deploy]
test:
stage: test
image: node:20
script:
- npm ci && npm test
build:
stage: build
image: docker:latest
services: [docker:dind]
script:
- docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
deploy:production:
stage: deploy
script:
- kubectl set image deployment/app app=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
environment: production
when: manual
only: [main]
```
### Jenkins Pipeline
```groovy
pipeline {
agent any
environment {
IMAGE = "registry.example.com/app"
}
stages {
stage('Test') {
steps {
sh 'npm ci && npm test'
junit 'reports/junit.xml'
}
}
stage('Build') {
steps {
script {
docker.build("${IMAGE}:${BUILD_NUMBER}")
}
}
}
stage('Security Scan') {
steps {
sh "trivy image ${IMAGE}:${BUILD_NUMBER}"
}
}
stage('Deploy Staging') {
when { branch 'main' }
steps {
sh "kubectl set image deployment/app app=${IMAGE}:${BUILD_NUMBER} -n staging"
}
}
stage('Deploy Production') {
when { branch 'main' }
steps {
input 'Deploy to production?'
sh "kubectl set image deployment/app app=${IMAGE}:${BUILD_NUMBER} -n production"
}
}
}
post {
failure {
slackSend color: 'danger', message: "Build failed: ${JOB_NAME}"
}
}
}
```
## Build Optimization
### Multi-stage Docker Build
```dockerfile
FROM node:20 AS deps
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
FROM node:20 AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build
FROM node:20-slim AS runner
WORKDIR /app
ENV NODE_ENV production
COPY --from=deps /app/node_modules ./node_modules
COPY --from=builder /app/dist ./dist
USER node
CMD ["node", "dist/main.js"]
```
### Parallel Testing
```yaml
# CircleCI
version: 2.1
jobs:
test:
parallelism: 4
docker:
- image: cimg/node:20
steps:
- checkout
- run: npm ci
- run: |
TESTS=$(circleci tests glob "test/**/*.js" | circleci tests split)
npm test $TESTS
```
## Release Orchestration
```bash
#!/bin/bash
# release.sh - Multi-service coordinated release
VERSION=$1
SERVICES=(auth api worker frontend)
echo "Release: $VERSION"
# Create release branches
for svc in "${SERVICES[@]}"; do
gh api repos/org/$svc/git/refs -f ref=refs/heads/release/$VERSION -f sha=$(git rev-parse main)
done
# Trigger builds
for svc in "${SERVICES[@]}"; do
gh workflow run ci.yml --repo org/$svc --ref release/$VERSION
done
# Wait for completion
for svc in "${SERVICES[@]}"; do
gh run watch --repo org/$svc $(gh run list --repo org/$svc -L1 -q '.[0].databaseId')
done
# Deploy to staging
kubectl apply -f staging/release-$VERSION.yaml
# Smoke tests
./scripts/smoke-test.sh staging
echo "✓ Release $VERSION ready for production"
```
## Dependency Management
### Renovate Auto-Update
```json
{
"extends": ["config:base"],
"packageRules": [
{
"matchUpdateTypes": ["minor", "patch"],
"automerge": true
},
{
"matchDepTypes": ["devDependencies"],
"automerge": true
}
],
"schedule": ["before 6am on Monday"],
"prConcurrentLimit": 5
}
```
## Build Optimization
### Build Caching Strategy
```yaml
# GitHub Actions: Multi-layer caching
- name: Cache dependencies
uses: actions/cache@v3
with:
path: |
~/.npm
~/.cache
node_modules
key: ${{ runner.os }}-deps-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-deps-
- name: Cache Docker layers
uses: docker/build-push-action@v4
with:
context: .
cache-from: type=gha
cache-to: type=gha,mode=max
```
### Parallel CI Pipeline
```yaml
# Multi-platform builds in parallel
name: Build
on: [push]
jobs:
test:
strategy:
matrix:
node: [18, 20, 22]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node }}
- run: npm ci && npm test
build-images:
needs: test
strategy:
matrix:
platform: [linux/amd64, linux/arm64]
runs-on: ubuntu-latest
steps:
- uses: docker/build-push-action@v4
with:
platforms: ${{ matrix.platform }}
tags: app:${{ github.sha }}
```
## Multi-Service Release Orchestration
```yaml
# release-coordinator.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: release-v2.5.0
spec:
template:
spec:
containers:
- name: coordinator
image: release-bot:latest
env:
- name: RELEASE_VERSION
value: "v2.5.0"
- name: SERVICES
value: "auth,api,worker,frontend"
command:
- /bin/bash
- -c
- |
# Deploy in dependency order
for svc in auth api worker frontend; do
echo "Deploying $svc..."
kubectl set image deploy/$svc \
$svc=registry.io/$svc:$RELEASE_VERSION
kubectl rollout status deploy/$svc --timeout=5m
# Health check
kubectl run test-$svc --rm -i --restart=Never \
--image=curlimages/curl -- \
curl -f http://$svc/health
echo "$svc deployed successfully"
done
```
## Advanced Artifact Management
```bash
#!/bin/bash
# artifact-scanner.sh - Scan before promotion
IMAGE=$1
SEVERITY=${2:-HIGH}
# Vulnerability scan
trivy image --severity $SEVERITY --exit-code 1 $IMAGE
# License compliance
syft $IMAGE -o json | \
jq '.artifacts[].licenses[] | select(.value |
contains("GPL") or contains("AGPL"))' && \
echo "License violation detected" && exit 1
# SBOM generation
syft $IMAGE -o spdx-json > sbom-$(basename $IMAGE).spdx.json
# Sign artifact
cosign sign --key cosign.key $IMAGE
# Promote
docker tag $IMAGE $IMAGE-approved
docker push $IMAGE-approved
echo "Artifact $IMAGE approved and promoted"
```
## Zero-Downtime Database Migrations
```python
# migrations/release_v2.5.py
from alembic import op
import sqlalchemy as sa
def upgrade():
# Step 1: Add new column (nullable)
op.add_column('users',
sa.Column('email_verified', sa.Boolean(), nullable=True))
# Step 2: Backfill data (in batches)
connection = op.get_bind()
connection.execute("""
UPDATE users SET email_verified = true
WHERE email IS NOT NULL
LIMIT 1000
""")
# Repeat until complete (or use background job)
# Step 3: Make non-nullable (in next release)
# op.alter_column('users', 'email_verified', nullable=False)
def downgrade():
op.drop_column('users', 'email_verified')
```
## Release Metrics Dashboard
```yaml
# Grafana dashboard for release metrics
apiVersion: v1
kind: ConfigMap
metadata:
name: release-dashboard
data:
dashboard.json: |
{
"panels": [
{
"title": "Deployment Frequency",
"targets": [{
"expr": "count_over_time(deployment_completed[1d])"
}]
},
{
"title": "Lead Time",
"targets": [{
"expr": "histogram_quantile(0.95, commit_to_deploy_seconds_bucket)"
}]
},
{
"title": "Change Failure Rate",
"targets": [{
"expr": "sum(rate(deployment_failed[1h])) / sum(rate(deployment_total[1h]))"
}]
},
{
"title": "Active Releases",
"targets": [{
"expr": "count(release_in_progress == 1)"
}]
}
]
}
```
## Best Practices
- Version artifacts with immutable tags
- Implement retention policies
- Use progressive delivery for high-risk changes
- Automate security scanning
- Maintain deployment audit trails
- Enable easy rollbacks
- Monitor deployment metrics
- Use feature flags for flexibility
- Cache aggressively for fast builds
- Parallelize test and build jobs
- Coordinate multi-service releases
- Generate and track SBOMs
- Sign artifacts for supply chain security
- Automate dependency updates
- Track DORA metrics continuously
@@ -0,0 +1,141 @@
# Terraform Infrastructure as Code
## AWS ECS Fargate Setup
```hcl
terraform {
required_providers {
aws = { source = "hashicorp/aws", version = "~> 5.0" }
}
backend "s3" {
bucket = "terraform-state"
key = "app/terraform.tfstate"
region = "us-east-1"
}
}
resource "aws_ecs_cluster" "main" {
name = "app-cluster"
setting {
name = "containerInsights"
value = "enabled"
}
}
resource "aws_ecs_task_definition" "app" {
family = "app"
network_mode = "awsvpc"
requires_compatibilities = ["FARGATE"]
cpu = "256"
memory = "512"
execution_role_arn = aws_iam_role.ecs_execution.arn
container_definitions = jsonencode([{
name = "app"
image = "${var.ecr_repository}:${var.image_tag}"
portMappings = [{ containerPort = 3000 }]
logConfiguration = {
logDriver = "awslogs"
options = {
awslogs-group = aws_cloudwatch_log_group.app.name
awslogs-region = var.region
awslogs-stream-prefix = "app"
}
}
secrets = [
{ name = "DATABASE_URL", valueFrom = aws_ssm_parameter.db_url.arn }
]
}])
}
resource "aws_ecs_service" "app" {
name = "app"
cluster = aws_ecs_cluster.main.id
task_definition = aws_ecs_task_definition.app.arn
desired_count = 2
launch_type = "FARGATE"
network_configuration {
subnets = var.private_subnets
security_groups = [aws_security_group.app.id]
}
load_balancer {
target_group_arn = aws_lb_target_group.app.arn
container_name = "app"
container_port = 3000
}
}
```
## AWS RDS PostgreSQL
```hcl
resource "aws_db_instance" "postgres" {
identifier = "app-db"
engine = "postgres"
engine_version = "16.1"
instance_class = "db.t3.micro"
allocated_storage = 20
storage_encrypted = true
db_name = "app"
username = "admin"
password = var.db_password
vpc_security_group_ids = [aws_security_group.db.id]
db_subnet_group_name = aws_db_subnet_group.main.name
backup_retention_period = 7
skip_final_snapshot = false
final_snapshot_identifier = "app-db-final"
tags = { Environment = var.environment }
}
```
## Variables and Outputs
```hcl
# variables.tf
variable "environment" {
type = string
description = "Environment name"
}
variable "region" {
type = string
default = "us-east-1"
}
# outputs.tf
output "ecs_cluster_arn" {
value = aws_ecs_cluster.main.arn
}
output "alb_dns_name" {
value = aws_lb.main.dns_name
}
```
## Best Practices
| Practice | Implementation |
|----------|----------------|
| State locking | S3 backend with DynamoDB |
| Secrets | Use AWS Secrets Manager / SSM |
| Modules | Reusable components |
| Workspaces | Environment separation |
| Tagging | Consistent resource tags |
| Validation | `terraform validate`, `tflint` |
## Common Commands
```bash
terraform init
terraform plan -out=tfplan
terraform apply tfplan
terraform destroy
terraform state list
terraform import aws_instance.app i-1234567890
```
+404
View File
@@ -0,0 +1,404 @@
---
name: reverse-proxy
description: Configure nginx and Traefik as reverse proxies. Implement SSL termination and routing. Use when setting up application gateways.
license: MIT
metadata:
author: devops-skills
version: "1.0"
---
# Reverse Proxy
Configure reverse proxies to route traffic, terminate TLS, enforce rate limits, and serve as the gateway between clients and backend services.
## When to Use
- Routing traffic from a public domain to one or more backend services.
- Terminating TLS at the edge and forwarding plain HTTP to backends.
- Adding rate limiting, CORS, security headers, and access control.
- Consolidating multiple services under a single domain with path-based routing.
- Handling WebSocket upgrades, gRPC proxying, or HTTP/2 passthrough.
## Prerequisites
- Backend service(s) running on known host:port.
- TLS certificate (Let's Encrypt, ACM, or self-signed for development).
- nginx 1.25+ or Traefik 3.x installed.
- DNS record pointing the domain to the proxy server.
## nginx Reverse Proxy
### Basic HTTPS Proxy with Redirect
```nginx
# /etc/nginx/sites-available/app.example.com
server {
listen 80;
server_name app.example.com;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl http2;
server_name app.example.com;
# TLS configuration
ssl_certificate /etc/letsencrypt/live/app.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/app.example.com/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
# Security headers
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;
add_header X-Frame-Options DENY always;
add_header X-Content-Type-Options nosniff always;
add_header Referrer-Policy strict-origin-when-cross-origin always;
# Proxy to backend
location / {
proxy_pass http://127.0.0.1:3000;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Timeouts
proxy_connect_timeout 5s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
# Buffering
proxy_buffering on;
proxy_buffer_size 4k;
proxy_buffers 8 4k;
}
}
```
### Path-Based Routing to Multiple Services
```nginx
server {
listen 443 ssl http2;
server_name app.example.com;
ssl_certificate /etc/letsencrypt/live/app.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/app.example.com/privkey.pem;
# Frontend SPA
location / {
proxy_pass http://127.0.0.1:3000;
proxy_set_header Host $host;
}
# API backend
location /api/ {
proxy_pass http://127.0.0.1:8080/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_read_timeout 120s;
}
# WebSocket endpoint
location /ws/ {
proxy_pass http://127.0.0.1:8080;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_read_timeout 86400s; # 24h for long-lived connections
}
# Static assets with caching
location /static/ {
alias /var/www/static/;
expires 30d;
add_header Cache-Control "public, immutable";
}
}
```
### Rate Limiting
```nginx
# Define rate limit zones in http block
http {
# 10 requests/second per IP
limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s;
# 1 request/second for login
limit_req_zone $binary_remote_addr zone=login_limit:10m rate=1r/s;
# Connection limit per IP
limit_conn_zone $binary_remote_addr zone=conn_limit:10m;
}
server {
listen 443 ssl http2;
server_name app.example.com;
# Apply rate limit to API
location /api/ {
limit_req zone=api_limit burst=20 nodelay;
limit_req_status 429;
proxy_pass http://127.0.0.1:8080;
}
# Strict rate limit on auth endpoints
location /api/auth/ {
limit_req zone=login_limit burst=5;
limit_req_status 429;
proxy_pass http://127.0.0.1:8080;
}
# Connection limit
location / {
limit_conn conn_limit 100;
proxy_pass http://127.0.0.1:3000;
}
}
```
### Gzip and Brotli Compression
```nginx
http {
gzip on;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml image/svg+xml;
gzip_min_length 256;
gzip_vary on;
gzip_proxied any;
gzip_comp_level 5;
# Brotli (requires ngx_brotli module)
# brotli on;
# brotli_types text/plain text/css application/json application/javascript text/xml application/xml image/svg+xml;
# brotli_comp_level 6;
}
```
### Let's Encrypt with Certbot
```bash
# Install certbot with nginx plugin
sudo apt install certbot python3-certbot-nginx
# Obtain and install certificate
sudo certbot --nginx -d app.example.com -d www.example.com
# Auto-renewal is configured via systemd timer
sudo systemctl status certbot.timer
# Manual renewal test
sudo certbot renew --dry-run
```
## Traefik Reverse Proxy
### Static Configuration
```yaml
# traefik.yml
entryPoints:
web:
address: ":80"
http:
redirections:
entryPoint:
to: websecure
scheme: https
websecure:
address: ":443"
certificatesResolvers:
letsencrypt:
acme:
email: admin@example.com
storage: /letsencrypt/acme.json
httpChallenge:
entryPoint: web
providers:
docker:
exposedByDefault: false
file:
directory: /etc/traefik/dynamic/
api:
dashboard: true
insecure: false
log:
level: INFO
accessLog:
filePath: /var/log/traefik/access.log
```
### Dynamic Configuration (File Provider)
```yaml
# /etc/traefik/dynamic/services.yml
http:
routers:
app:
rule: "Host(`app.example.com`)"
entryPoints:
- websecure
service: app
tls:
certResolver: letsencrypt
middlewares:
- security-headers
- rate-limit
api:
rule: "Host(`app.example.com`) && PathPrefix(`/api`)"
entryPoints:
- websecure
service: api
tls:
certResolver: letsencrypt
services:
app:
loadBalancer:
servers:
- url: "http://127.0.0.1:3000"
healthCheck:
path: /health
interval: 10s
timeout: 3s
api:
loadBalancer:
servers:
- url: "http://127.0.0.1:8080"
healthCheck:
path: /api/health
interval: 10s
timeout: 3s
middlewares:
security-headers:
headers:
stsSeconds: 63072000
stsIncludeSubdomains: true
frameDeny: true
contentTypeNosniff: true
browserXssFilter: true
referrerPolicy: strict-origin-when-cross-origin
rate-limit:
rateLimit:
average: 100
burst: 50
period: 1m
```
### Traefik with Docker Labels
```yaml
# docker-compose.yml
version: "3.8"
services:
traefik:
image: traefik:v3.0
ports:
- "80:80"
- "443:443"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- ./traefik.yml:/etc/traefik/traefik.yml:ro
- letsencrypt:/letsencrypt
frontend:
image: my-frontend:latest
labels:
- "traefik.enable=true"
- "traefik.http.routers.frontend.rule=Host(`app.example.com`)"
- "traefik.http.routers.frontend.tls.certresolver=letsencrypt"
- "traefik.http.services.frontend.loadbalancer.server.port=3000"
api:
image: my-api:latest
labels:
- "traefik.enable=true"
- "traefik.http.routers.api.rule=Host(`app.example.com`) && PathPrefix(`/api`)"
- "traefik.http.routers.api.tls.certresolver=letsencrypt"
- "traefik.http.services.api.loadbalancer.server.port=8080"
- "traefik.http.routers.api.middlewares=api-ratelimit"
- "traefik.http.middlewares.api-ratelimit.ratelimit.average=50"
- "traefik.http.middlewares.api-ratelimit.ratelimit.burst=25"
volumes:
letsencrypt:
```
## nginx Testing and Management
```bash
# Test configuration syntax
sudo nginx -t
# Reload without downtime
sudo nginx -s reload
# View active connections
sudo nginx -s status
# Check which config file is active
nginx -V 2>&1 | grep -o '\-\-conf-path=[^ ]*'
# Monitor access logs
tail -f /var/log/nginx/access.log
# Monitor error logs
tail -f /var/log/nginx/error.log
```
## IP Allowlisting and Geoblocking
```nginx
# Allow only specific IPs (admin panel)
location /admin/ {
allow 203.0.113.0/24;
allow 198.51.100.5;
deny all;
proxy_pass http://127.0.0.1:3000;
}
# Block by country (requires GeoIP2 module)
# geoip2 /usr/share/GeoIP/GeoLite2-Country.mmdb {
# auto_reload 60m;
# $geoip2_data_country_iso_code country iso_code;
# }
# if ($geoip2_data_country_iso_code = "XX") {
# return 403;
# }
```
## Troubleshooting
| Symptom | Cause | Fix |
|---------|-------|-----|
| 502 Bad Gateway | Backend not running or unreachable | Verify backend is listening; check `proxy_pass` URL |
| 504 Gateway Timeout | Backend too slow | Increase `proxy_read_timeout`; check backend performance |
| Mixed content warnings | `X-Forwarded-Proto` not set | Add `proxy_set_header X-Forwarded-Proto $scheme` |
| WebSocket disconnects after 60s | Default proxy timeout expires | Set `proxy_read_timeout 86400s` for WebSocket locations |
| Rate limit hits legitimate users | Zone rate too aggressive | Increase `rate` or `burst` values; use different zones per endpoint |
| Let's Encrypt renewal fails | Port 80 blocked or wrong server block | Ensure `.well-known/acme-challenge/` is accessible |
| Traefik shows 404 for all routes | Docker labels not detected | Verify Docker socket is mounted; check `exposedByDefault` setting |
| TLS handshake failure | Certificate chain incomplete | Include intermediate certificates in `ssl_certificate` |
## Related Skills
- [load-balancing](../load-balancing/) - Multi-backend traffic distribution
- [cdn-setup](../cdn-setup/) - CDN in front of reverse proxy
- [dns-management](../dns-management/) - DNS records for proxy domains
- [service-mesh](../service-mesh/) - Service-level routing in Kubernetes
@@ -0,0 +1,7 @@
{
"version": 1,
"registry": "https://clawhub.ai",
"slug": "sysadmin-toolbox",
"installedVersion": "1.1.0",
"installedAt": 1772232079431
}
+102
View File
@@ -0,0 +1,102 @@
---
name: sysadmin-toolbox
description: "Tool discovery and shell one-liner reference for sysadmin, DevOps, and security tasks. AUTO-CONSULT this skill when the user is: troubleshooting network issues, debugging processes, analyzing logs, working with SSL/TLS, managing DNS, testing HTTP endpoints, auditing security, working with containers, writing shell scripts, or asks 'what tool should I use for X'. Source: github.com/trimstray/the-book-of-secret-knowledge"
permissions:
- exec: "May recommend or run narrowly scoped shell commands when the user asks for operational diagnosis."
- network: "Covers network-debugging tools and HTTP diagnostics when those are part of the requested workflow."
---
# Sysadmin Toolbox
Curated tool recommendations and practical shell one-liners for operational work.
## When to Auto-Consult
Load relevant references when user is:
- Debugging network connectivity, ports, traffic
- Troubleshooting DNS or SSL/TLS
- Analyzing processes, memory, disk usage
- Working with logs or system diagnostics
- Writing shell scripts or one-liners
- Asking "what's a good tool for..."
- Doing security audits or pentesting
- Working with containers/Docker/K8s
## Bundled Guides
| File | Use When |
|------|----------|
| `shell-oneliners.md` in this skill's `references` folder | Need practical commands for: terminal, networking, SSL, curl, ssh, tcpdump, git, awk, sed, grep, find |
| `cli-tools.md` in this skill's `references` folder | Recommending CLI tools: shells, file managers, network utils, databases, security tools |
| `web-tools.md` in this skill's `references` folder | Web-based tools: SSL checkers, DNS lookup, performance testing, OSINT, scanners |
| `security-tools.md` in this skill's `references` folder | Pentesting, vulnerability scanning, exploit databases, CTF resources |
| `shell-tricks.md` in this skill's `references` folder | Shell scripting patterns and tricks |
## Safety Boundaries
- Do not run destructive system commands, privilege-escalation steps, or offensive tooling unless the user explicitly asked for that scope.
- Do not scan hosts, domains, or networks the user does not control or have permission to assess.
- Do not assume a command is safe to paste into production without explaining what it does.
- Do not persist logs, captures, or credentials outside the user's requested troubleshooting workflow.
## Quick Tool Index
### Network Debugging
- `mtr` - traceroute + ping combined
- `tcpdump` / `tshark` - packet capture
- `netstat` / `ss` - connection monitoring
- `nmap` - port scanning
- `curl` / `httpie` - HTTP testing
### DNS
- `dig` / `host` - DNS queries
- `dnsdiag` - DNS diagnostics
- `subfinder` / `amass` - subdomain enumeration
### SSL/TLS
- `openssl` - certificate inspection
- `testssl.sh` - TLS testing
- `sslyze` - SSL scanning
- `certbot` - Let's Encrypt
### Process/System
- `htop` / `btop` - process monitoring
- `strace` / `ltrace` - syscall/library tracing
- `lsof` - open files/connections
- `ncdu` - disk usage
### Log Analysis
- `lnav` - log navigator
- `GoAccess` - web log analyzer
- `angle-grinder` - log slicing
### Containers
- `dive` - Docker image analysis
- `ctop` - container top
- `lazydocker` - Docker TUI
## Keeping Current
References auto-refresh weekly (Sundays 5am ET) from the upstream repo:
```bash
~/clawd-duke-leto/skills/sysadmin-toolbox/scripts/refresh.sh
```
Manual refresh anytime:
```bash
./scripts/refresh.sh [skill-dir]
```
## Example Queries → Actions
**"Why is this port not responding?"**
→ Load shell-oneliners.md, search for netstat/ss/lsof commands
**"What's a good tool for testing SSL?"**
→ Load cli-tools.md SSL section, recommend testssl.sh or sslyze
**"Show me how to find large files"**
→ Load shell-oneliners.md, search for find/ncdu/du commands
**"I need to debug DNS resolution"**
→ Load shell-oneliners.md dig section + recommend dnsdiag from cli-tools.md
@@ -0,0 +1,6 @@
{
"ownerId": "kn73ft5fmjrf1h9n78fc61tgfs7yp8ez",
"slug": "sysadmin-toolbox",
"version": "1.1.0",
"publishedAt": 1768843243799
}
@@ -0,0 +1,272 @@
#### CLI Tools &nbsp;[<sup>[TOC]</sup>](#anger-table-of-contents)
##### :black_small_square: Shells
<p>
&nbsp;&nbsp; <a href="https://www.gnu.org/software/bash/"><b>GNU Bash</b></a> - is an sh-compatible shell that incorporates useful features from the Korn shell and C shell.<br>
&nbsp;&nbsp; <a href="https://www.zsh.org/"><b>Zsh</b></a> - is a shell designed for interactive use, although it is also a powerful scripting language.<br>
&nbsp;&nbsp; <a href="https://tcl-lang.org/"><b>tclsh</b></a> - is a very powerful cross-platform shell, suitable for a huge range of uses.<br>
&nbsp;&nbsp; <a href="https://github.com/Bash-it/bash-it"><b>bash-it</b></a> - is a framework for using, developing and maintaining shell scripts and custom commands.<br>
&nbsp;&nbsp; <a href="https://ohmyz.sh/"><b>Oh My ZSH!</b></a> - is the best framework for managing your Zsh configuration.<br>
&nbsp;&nbsp; <a href="https://github.com/oh-my-fish/oh-my-fish"><b>Oh My Fish</b></a> - the Fishshell framework.<br>
&nbsp;&nbsp; <a href="https://github.com/starship/starship"><b>Starship</b></a> - the cross-shell prompt written in Rust.<br>
&nbsp;&nbsp; <a href="https://github.com/romkatv/powerlevel10k"><b>powerlevel10k</b></a> - is a fast reimplementation of Powerlevel9k ZSH theme.<br>
</p>
##### :black_small_square: Shell plugins
<p>
&nbsp;&nbsp; <a href="https://github.com/rupa/z"><b>z</b></a> - tracks the folder you use the most and allow you to jump, without having to type the whole path.<br>
&nbsp;&nbsp; <a href="https://github.com/junegunn/fzf"><b>fzf</b></a> - is a general-purpose command-line fuzzy finder.<br>
&nbsp;&nbsp; <a href="https://github.com/zsh-users/zsh-autosuggestions"><b>zsh-autosuggestions</b></a> - Fish-like autosuggestions for Zsh.<br>
&nbsp;&nbsp; <a href="https://github.com/zsh-users/zsh-syntax-highlighting"><b>zsh-syntax-highlighting</b></a> - Fish shell like syntax highlighting for Zsh.<br>
&nbsp;&nbsp; <a href="https://github.com/unixorn/awesome-zsh-plugins"><b>Awesome ZSH Plugins</b></a> - A list of frameworks, plugins, themes and tutorials for ZSH.<br>
</p>
##### :black_small_square: Managers
<p>
&nbsp;&nbsp; <a href="https://midnight-commander.org/"><b>Midnight Commander</b></a> - is a visual file manager, licensed under GNU General Public License.<br>
&nbsp;&nbsp; <a href="https://github.com/ranger/ranger"><b>ranger</b></a> - is a VIM-inspired filemanager for the console.<br>
&nbsp;&nbsp; <a href="https://github.com/jarun/nnn"><b>nnn</b></a> - is a tiny, lightning fast, feature-packed file manager.<br>
&nbsp;&nbsp; <a href="https://www.gnu.org/software/screen/"><b>screen</b></a> - is a full-screen window manager that multiplexes a physical terminal.<br>
&nbsp;&nbsp; <a href="https://github.com/tmux/tmux/wiki"><b>tmux</b></a> - is a terminal multiplexer, lets you switch easily between several programs in one terminal.<br>
&nbsp;&nbsp; <a href="https://github.com/peikk0/tmux-cssh"><b>tmux-cssh</b></a> - is a tool to set comfortable and easy to use functionality tmux-sessions.<br>
</p>
##### :black_small_square: Text editors
<p>
&nbsp;&nbsp; <a href="http://ex-vi.sourceforge.net/"><b>vi</b></a> - is one of the most common text editors on Unix.<br>
&nbsp;&nbsp; <a href="https://www.vim.org/"><b>vim</b></a> - is a highly configurable text editor.<br>
&nbsp;&nbsp; <a href="https://www.gnu.org/software/emacs/"><b>emacs</b></a> - is an extensible, customizable, free/libre text editor, and more.<br>
&nbsp;&nbsp; <a href="https://github.com/zyedidia/micro"><b>micro</b></a> - is a modern and intuitive terminal-based text editor.<br>
&nbsp;&nbsp; <a href="https://neovim.io/"><b>neovim</b></a> - is a free open source, powerful, extensible and usable code editor.<br>
&nbsp;&nbsp; <a href="https://www.spacemacs.org/"><b>spacemacs</b></a> - a community-driven Emacs distribution.<br>
&nbsp;&nbsp; <a href="https://spacevim.org/"><b>spacevim</b></a> - a community-driven vim distribution.<br>
</p>
##### :black_small_square: Files and directories
<p>
&nbsp;&nbsp; <a href="https://github.com/sharkdp/fd"><b>fd</b></a> - is a simple, fast and user-friendly alternative to find.<br>
&nbsp;&nbsp; <a href="https://dev.yorhel.nl/ncdu"><b>ncdu</b></a> - is an easy to use, fast disk usage analyzer.<br>
</p>
##### :black_small_square: Network
<p>
&nbsp;&nbsp; <a href="https://www.putty.org/"><b>PuTTY</b></a> - is an SSH and telnet client, developed originally by Simon Tatham.<br>
&nbsp;&nbsp; <a href="https://mosh.org/"><b>Mosh</b></a> - is a SSH wrapper designed to keep a SSH session alive over a volatile connection.<br>
&nbsp;&nbsp; <a href="https://eternalterminal.dev/"><b>Eternal Terminal</b></a> - enables mouse-scrolling and tmux commands inside the SSH session.<br>
&nbsp;&nbsp; <a href="https://nmap.org/"><b>nmap</b></a> - is a free and open source (license) utility for network discovery and security auditing.<br>
&nbsp;&nbsp; <a href="https://github.com/zmap/zmap"><b>zmap</b></a> - is a fast single packet network scanner designed for Internet-wide network surveys.<br>
&nbsp;&nbsp; <a href="https://github.com/RustScan/RustScan"><b>Rust Scan</b></a> - to find all open ports faster than Nmap.<br>
&nbsp;&nbsp; <a href="https://github.com/robertdavidgraham/masscan"><b>masscan</b></a> - is the fastest Internet port scanner, spews SYN packets asynchronously.<br>
&nbsp;&nbsp; <a href="https://github.com/gvb84/pbscan"><b>pbscan</b></a> - is a faster and more efficient stateless SYN scanner and banner grabber.<br>
&nbsp;&nbsp; <a href="http://www.hping.org/"><b>hping</b></a> - is a command-line oriented TCP/IP packet assembler/analyzer.<br>
&nbsp;&nbsp; <a href="https://github.com/traviscross/mtr"><b>mtr</b></a> - is a tool that combines the functionality of the 'traceroute' and 'ping' programs in a single tool.<br>
&nbsp;&nbsp; <a href="https://github.com/mehrdadrad/mylg"><b>mylg</b></a> - utility which combines the functions of the different network probes in one diagnostic tool.<br>
&nbsp;&nbsp; <a href="http://netcat.sourceforge.net/"><b>netcat</b></a> - utility which reads and writes data across network connections, using the TCP/IP protocol.<br>
&nbsp;&nbsp; <a href="http://www.dest-unreach.org/socat/"><b>socat</b></a> - utility which transfers data between two objects.<br>
&nbsp;&nbsp; <a href="https://www.tcpdump.org/"><b>tcpdump</b></a> - is a powerful command-line packet analyzer.<br>
&nbsp;&nbsp; <a href="https://www.wireshark.org/docs/man-pages/tshark.html"><b>tshark</b></a> - is a tool that allows us to dump and analyze network traffic (wireshark cli).<br>
&nbsp;&nbsp; <a href="https://termshark.io/"><b>Termshark</b></a> - is a simple terminal user-interface for tshark.<br>
&nbsp;&nbsp; <a href="https://github.com/jpr5/ngrep"><b>ngrep</b></a> - is like GNU grep applied to the network layer.<br>
&nbsp;&nbsp; <a href="http://netsniff-ng.org/"><b>netsniff-ng</b></a> - is a Swiss army knife for your daily Linux network plumbing if you will.<br>
&nbsp;&nbsp; <a href="https://github.com/mechpen/sockdump"><b>sockdump</b></a> - dump unix domain socket traffic.<br>
&nbsp;&nbsp; <a href="https://github.com/google/stenographer"><b>stenographer</b></a> - is a packet capture solution which aims to quickly spool all packets to disk.<br>
&nbsp;&nbsp; <a href="https://github.com/sachaos/tcpterm"><b>tcpterm</b></a> - visualize packets in TUI.<br>
&nbsp;&nbsp; <a href="https://github.com/tgraf/bmon"><b>bmon</b></a> - is a monitoring and debugging tool to capture networking related statistics and prepare them visually.<br>
&nbsp;&nbsp; <a href="http://iptraf.seul.org/2.6/manual.html#installation"><b>iptraf-ng</b></a> - is a console-based network monitoring program for Linux that displays information about IP traffic.<br>
&nbsp;&nbsp; <a href="https://github.com/vergoh/vnstat"><b>vnstat</b></a> - is a network traffic monitor for Linux and BSD.<br>
&nbsp;&nbsp; <a href="https://iperf.fr/"><b>iPerf3</b></a> - is a tool for active measurements of the maximum achievable bandwidth on IP networks.<br>
&nbsp;&nbsp; <a href="https://github.com/Microsoft/Ethr"><b>ethr</b></a> - is a Network Performance Measurement Tool for TCP, UDP & HTTP.<br>
&nbsp;&nbsp; <a href="https://github.com/jwbensley/Etherate"><b>Etherate</b></a> - is a Linux CLI based Ethernet and MPLS traffic testing tool.<br>
&nbsp;&nbsp; <a href="https://github.com/mpolden/echoip"><b>echoip</b></a> - is a IP address lookup service.<br>
&nbsp;&nbsp; <a href="https://github.com/troglobit/nemesis"><b>Nemesis</b></a> - packet manipulation CLI tool; craft and inject packets of several protocols.<br>
&nbsp;&nbsp; <a href="https://github.com/packetfu/packetfu"><b>packetfu</b></a> - a mid-level packet manipulation library for Ruby.<br>
&nbsp;&nbsp; <a href="https://scapy.net/"><b>Scapy</b></a> - packet manipulation library; forge, send, decode, capture packets of a wide number of protocols.<br>
&nbsp;&nbsp; <a href="https://github.com/SecureAuthCorp/impacket"><b>impacket</b></a> - is a collection of Python classes for working with network protocols.<br>
&nbsp;&nbsp; <a href="https://github.com/arthepsy/ssh-audit"><b>ssh-audit</b></a> - is a tool for SSH server auditing.<br>
&nbsp;&nbsp; <a href="https://aria2.github.io/"><b>aria2</b></a> - is a lightweight multi-protocol & multi-source command-line download utility.<br>
&nbsp;&nbsp; <a href="https://github.com/x-way/iptables-tracer"><b>iptables-tracer</b></a> - observe the path of packets through the iptables chains.<br>
&nbsp;&nbsp; <a href="https://github.com/proabiral/inception"><b>inception</b></a> - a highly configurable tool to check for whatever you like against any number of hosts.<br>
&nbsp;&nbsp; <a href="https://mremoteng.org/"><b>mRemoteNG</b></a> - a fork of mRemote, multi-tabbed PuTTy on steroids!<br>
</p>
##### :black_small_square: Network (DNS)
<p>
&nbsp;&nbsp; <a href="https://github.com/farrokhi/dnsdiag"><b>dnsdiag</b></a> - is a DNS diagnostics and performance measurement tools.<br>
&nbsp;&nbsp; <a href="https://github.com/mschwager/fierce"><b>fierce</b></a> - is a DNS reconnaissance tool for locating non-contiguous IP space.<br>
&nbsp;&nbsp; <a href="https://github.com/subfinder/subfinder"><b>subfinder</b></a> - is a subdomain discovery tool that discovers valid subdomains for websites.<br>
&nbsp;&nbsp; <a href="https://github.com/aboul3la/Sublist3r"><b>sublist3r</b></a> - is a fast subdomains enumeration tool for penetration testers.<br>
&nbsp;&nbsp; <a href="https://github.com/OWASP/Amass"><b>amass</b></a> - is tool that obtains subdomain names by scraping data sources, crawling web archives, and more.<br>
&nbsp;&nbsp; <a href="https://github.com/google/namebench"><b>namebench</b></a> - provides personalized DNS server recommendations based on your browsing history.<br>
&nbsp;&nbsp; <a href="https://github.com/blechschmidt/massdns"><b>massdns</b></a> - is a high-performance DNS stub resolver for bulk lookups and reconnaissance.<br>
&nbsp;&nbsp; <a href="https://github.com/guelfoweb/knock"><b>knock</b></a> - is a tool to enumerate subdomains on a target domain through a wordlist.<br>
&nbsp;&nbsp; <a href="https://github.com/DNS-OARC/dnsperf"><b>dnsperf</b></a> - DNS performance testing tools.<br>
&nbsp;&nbsp; <a href="https://github.com/jedisct1/dnscrypt-proxy"><b>dnscrypt-proxy 2</b></a> - a flexible DNS proxy, with support for encrypted DNS protocols.<br>
&nbsp;&nbsp; <a href="https://github.com/dnsdb/dnsdbq"><b>dnsdbq</b></a> - API client providing access to passive DNS database systems.<br>
&nbsp;&nbsp; <a href="https://github.com/looterz/grimd"><b>grimd</b></a> - fast dns proxy, built to black-hole internet advertisements and malware servers.<br>
&nbsp;&nbsp; <a href="https://github.com/elceef/dnstwist"><b>dnstwist</b></a> - detect typosquatters, phishing attacks, fraud, and brand impersonation.<br>
</p>
##### :black_small_square: Network (HTTP)
<p>
&nbsp;&nbsp; <a href="https://curl.haxx.se/"><b>curl</b></a> - is a command line tool and library for transferring data with URLs.<br>
&nbsp;&nbsp; <a href="https://gitlab.com/davidjpeacock/kurly"><b>kurly</b></a> - is an alternative to the widely popular curl program, written in Golang.<br>
&nbsp;&nbsp; <a href="https://github.com/jakubroztocil/httpie"><b>HTTPie</b></a> - is an user-friendly HTTP client.<br>
&nbsp;&nbsp; <a href="https://github.com/asciimoo/wuzz"><b>wuzz</b></a> - is an interactive cli tool for HTTP inspection.<br>
&nbsp;&nbsp; <a href="https://github.com/summerwind/h2spec"><b>h2spec</b></a> - is a conformance testing tool for HTTP/2 implementation.<br>
&nbsp;&nbsp; <a href="https://github.com/gildasio/h2t"><b>h2t</b></a> - is a simple tool to help sysadmins to hardening their websites.<br>
&nbsp;&nbsp; <a href="https://github.com/trimstray/htrace.sh"><b>htrace.sh</b></a> - is a simple Swiss Army knife for http/https troubleshooting and profiling.<br>
&nbsp;&nbsp; <a href="https://github.com/reorx/httpstat"><b>httpstat</b></a> - is a tool that visualizes curl statistics in a way of beauty and clarity.<br>
&nbsp;&nbsp; <a href="https://github.com/gchaincl/httplab"><b>httplab</b></a> - is an interactive web server.<br>
&nbsp;&nbsp; <a href="https://lynx.browser.org/"><b>Lynx</b></a> - is a text browser for the World Wide Web.<br>
&nbsp;&nbsp; <a href="https://github.com/browsh-org/browsh/"><b>Browsh</b></a> - is a fully interactive, real-time, and modern text-based browser.<br>
&nbsp;&nbsp; <a href="https://github.com/dhamaniasad/HeadlessBrowsers"><b>HeadlessBrowsers</b></a> - a list of (almost) all headless web browsers in existence.<br>
&nbsp;&nbsp; <a href="https://httpd.apache.org/docs/2.4/programs/ab.html"><b>ab</b></a> - is a single-threaded command line tool for measuring the performance of HTTP web servers.<br>
&nbsp;&nbsp; <a href="https://www.joedog.org/siege-home/"><b>siege</b></a> - is an http load testing and benchmarking utility.<br>
&nbsp;&nbsp; <a href="https://github.com/wg/wrk"><b>wrk</b></a> - is a modern HTTP benchmarking tool capable of generating significant load.<br>
&nbsp;&nbsp; <a href="https://github.com/giltene/wrk2"><b>wrk2</b></a> - is a constant throughput, correct latency recording variant of wrk.<br>
&nbsp;&nbsp; <a href="https://github.com/tsenart/vegeta"><b>vegeta</b></a> - is a constant throughput, correct latency recording variant of wrk.<br>
&nbsp;&nbsp; <a href="https://github.com/codesenberg/bombardier"><b>bombardier</b></a> - is a fast cross-platform HTTP benchmarking tool written in Go.<br>
&nbsp;&nbsp; <a href="https://github.com/cmpxchg16/gobench"><b>gobench</b></a> - http/https load testing and benchmarking tool.<br>
&nbsp;&nbsp; <a href="https://github.com/rakyll/hey"><b>hey</b></a> - HTTP load generator, ApacheBench (ab) replacement, formerly known as rakyll/boom.<br>
&nbsp;&nbsp; <a href="https://github.com/tarekziade/boom"><b>boom</b></a> - is a script you can use to quickly smoke-test your web app deployment.<br>
&nbsp;&nbsp; <a href="https://github.com/shekyan/slowhttptest"><b>SlowHTTPTest</b></a> - is a tool that simulates some Application Layer Denial of Service attacks by prolonging HTTP.<br>
&nbsp;&nbsp; <a href="https://github.com/OJ/gobuster"><b>gobuster</b></a> - is a free and open source directory/file & DNS busting tool written in Go.<br>
&nbsp;&nbsp; <a href="https://github.com/ssllabs/ssllabs-scan"><b>ssllabs-scan</b></a> - command-line reference-implementation client for SSL Labs APIs.<br>
&nbsp;&nbsp; <a href="https://github.com/mozilla/http-observatory"><b>http-observatory</b></a> - Mozilla HTTP Observatory cli version.<br>
&nbsp;&nbsp; <a href="https://hurl.dev"><b>Hurl</b></a> - is a command line tool to run and test HTTP requests with plain text.<br>
</p>
##### :black_small_square: SSL
<p>
&nbsp;&nbsp; <a href="https://www.openssl.org/"><b>openssl</b></a> - is a robust, commercial-grade, and full-featured toolkit for the TLS and SSL protocols.<br>
&nbsp;&nbsp; <a href="https://gnutls.org/manual/html_node/gnutls_002dcli-Invocation.html"><b>gnutls-cli</b></a> - client program to set up a TLS connection to some other computer.<br>
&nbsp;&nbsp; <a href="https://github.com/nabla-c0d3/sslyze"><b>sslyze
</b></a> - fast and powerful SSL/TLS server scanning library.<br>
&nbsp;&nbsp; <a href="https://github.com/rbsec/sslscan"><b>sslscan</b></a> - tests SSL/TLS enabled services to discover supported cipher suites.<br>
&nbsp;&nbsp; <a href="https://github.com/drwetter/testssl.sh"><b>testssl.sh</b></a> - testing TLS/SSL encryption anywhere on any port.<br>
&nbsp;&nbsp; <a href="https://github.com/mozilla/cipherscan"><b>cipherscan</b></a> - a very simple way to find out which SSL ciphersuites are supported by a target.<br>
&nbsp;&nbsp; <a href="http://www.tarsnap.com/spiped.html"><b>spiped</b></a> - is a utility for creating symmetrically encrypted and authenticated pipes between socket addresses.<br>
&nbsp;&nbsp; <a href="https://github.com/certbot/certbot"><b>Certbot</b></a> - is EFF's tool to obtain certs from Let's Encrypt and (optionally) auto-enable HTTPS on your server.<br>
&nbsp;&nbsp; <a href="https://github.com/FiloSottile/mkcert"><b>mkcert</b></a> - simple zero-config tool to make locally trusted development certificates with any names you'd like.<br>
&nbsp;&nbsp; <a href="https://github.com/square/certstrap"><b>certstrap</b></a> - tools to bootstrap CAs, certificate requests, and signed certificates.<br>
&nbsp;&nbsp; <a href="https://github.com/yassineaboukir/sublert"><b>Sublert</b></a> - is a security and reconnaissance tool to automatically monitor new subdomains.<br>
&nbsp;&nbsp; <a href="https://github.com/trimstray/mkchain"><b>mkchain</b></a> - open source tool to help you build a valid SSL certificate chain.<br>
&nbsp;&nbsp; <a href="https://github.com/Matty9191/ssl-cert-check"><b>ssl-cert-check</b></a> - SSL Certification Expiration Checker.<br>
</p>
##### :black_small_square: Security
<p>
&nbsp;&nbsp; <a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/deployment_guide/ch-selinux"><b>SELinux</b></a> - provides a flexible Mandatory Access Control (MAC) system built into the Linux kernel.<br>
&nbsp;&nbsp; <a href="https://wiki.ubuntu.com/AppArmor"><b>AppArmor</b></a> - proactively protects the operating system and applications from external or internal threats.<br>
&nbsp;&nbsp; <a href="https://github.com/grapheneX/grapheneX"><b>grapheneX</b></a> - Automated System Hardening Framework.<br>
&nbsp;&nbsp; <a href="https://github.com/dev-sec/"><b>DevSec Hardening Framework</b></a> - Security + DevOps: Automatic Server Hardening.<br>
</p>
##### :black_small_square: Auditing Tools
<p>
&nbsp;&nbsp; <a href="https://www.ossec.net/"><b>ossec</b></a> - actively monitoring all aspects of system activity with file integrity monitoring.<br>
&nbsp;&nbsp; <a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security_guide/chap-system_auditing"><b>auditd</b></a> - provides a way to track security-relevant information on your system.<br>
&nbsp;&nbsp; <a href="https://www.nongnu.org/tiger/"><b>Tiger</b></a> - is a security tool that can be use both as a security audit and intrusion detection system.<br>
&nbsp;&nbsp; <a href="https://cisofy.com/lynis/"><b>Lynis</b></a> - battle-tested security tool for systems running Linux, macOS, or Unix-based operating system.<br>
&nbsp;&nbsp; <a href="https://github.com/rebootuser/LinEnum"><b>LinEnum</b></a> - scripted Local Linux Enumeration & Privilege Escalation Checks.<br>
&nbsp;&nbsp; <a href="https://github.com/installation/rkhunter"><b>Rkhunter</b></a> - scanner tool for Linux systems that scans backdoors, rootkits and local exploits on your systems.<br>
&nbsp;&nbsp; <a href="https://github.com/hasherezade/pe-sieve"><b>PE-sieve</b></a> - is a light-weight tool that helps to detect malware running on the system.<br>
&nbsp;&nbsp; <a href="https://github.com/carlospolop/privilege-escalation-awesome-scripts-suite"><b>PEASS</b></a> - privilege escalation tools for Windows and Linux/Unix and MacOS.<br>
</p>
##### :black_small_square: System Diagnostics/Debuggers
<p>
&nbsp;&nbsp; <a href="https://github.com/strace/strace"><b>strace</b></a> - diagnostic, debugging and instructional userspace utility for Linux.<br>
&nbsp;&nbsp; <a href="http://dtrace.org/blogs/about/"><b>DTrace</b></a> - is a performance analysis and troubleshooting tool.<br>
&nbsp;&nbsp; <a href="https://en.wikipedia.org/wiki/Ltrace"><b>ltrace</b></a> - is a library call tracer, used to trace calls made by programs to library functions.<br>
&nbsp;&nbsp; <a href="https://github.com/brainsmoke/ptrace-burrito"><b>ptrace-burrito</b></a> - is a friendly wrapper around ptrace.<br>
&nbsp;&nbsp; <a href="https://github.com/brendangregg/perf-tools"><b>perf-tools</b></a> - performance analysis tools based on Linux perf_events (aka perf) and ftrace.<br>
&nbsp;&nbsp; <a href="https://github.com/iovisor/bpftrace"><b>bpftrace</b></a> - high-level tracing language for Linux eBPF.<br>
&nbsp;&nbsp; <a href="https://github.com/draios/sysdig"><b>sysdig</b></a> - system exploration and troubleshooting tool with first class support for containers.<br>
&nbsp;&nbsp; <a href="http://www.valgrind.org/"><b>Valgrind</b></a> - is an instrumentation framework for building dynamic analysis tools.<br>
&nbsp;&nbsp; <a href="https://github.com/gperftools/gperftools"><b>gperftools</b></a> - high-performance multi-threaded malloc() implementation, plus some performance analysis tools.<br>
&nbsp;&nbsp; <a href="https://nicolargo.github.io/glances/"><b>glances</b></a> - cross-platform system monitoring tool written in Python.<br>
&nbsp;&nbsp; <a href="https://github.com/hishamhm/htop"><b>htop</b></a> - interactive text-mode process viewer for Unix systems. It aims to be a better 'top'.<br>
&nbsp;&nbsp; <a href="https://github.com/aristocratos/bashtop"><b>bashtop</b></a> - Linux resource monitor written in pure Bash.<br>
&nbsp;&nbsp; <a href="http://nmon.sourceforge.net/pmwiki.php"><b>nmon</b></a> - a single executable for performance monitoring and data analysis.<br>
&nbsp;&nbsp; <a href="https://www.atoptool.nl/"><b>atop</b></a> - ASCII performance monitor. Includes statistics for CPU, memory, disk, swap, network, and processes.<br>
&nbsp;&nbsp; <a href="https://en.wikipedia.org/wiki/Lsof"><b>lsof</b></a> - displays in its output information about files that are opened by processes.<br>
&nbsp;&nbsp; <a href="http://www.brendangregg.com/flamegraphs.html"><b>FlameGraph</b></a> - stack trace visualizer.<br>
&nbsp;&nbsp; <a href="https://github.com/zevv/lsofgraph"><b>lsofgraph</b></a> - convert Unix lsof output to a graph showing FIFO and UNIX interprocess communication.<br>
&nbsp;&nbsp; <a href="https://github.com/mozilla/rr"><b>rr</b></a> - is a lightweight tool for recording, replaying and debugging execution of applications.<br>
&nbsp;&nbsp; <a href="https://pcp.io/index.html"><b>Performance Co-Pilot</b></a> - a system performance analysis toolkit.<br>
&nbsp;&nbsp; <a href="https://github.com/sharkdp/hexyl"><b>hexyl</b></a> - a command-line hex viewer.<br>
&nbsp;&nbsp; <a href="https://github.com/p403n1x87/austin"><b>Austin</b></a> - Python frame stack sampler for CPython.<br>
</p>
##### :black_small_square: Log Analyzers
<p>
&nbsp;&nbsp; <a href="https://github.com/rcoh/angle-grinder"><b>angle-grinder</b></a> - slice and dice log files on the command line.<br>
&nbsp;&nbsp; <a href="https://lnav.org"><b>lnav</b></a> - log file navigator with search and automatic refresh.<br>
&nbsp;&nbsp; <a href="https://goaccess.io/"><b>GoAccess</b></a> - real-time web log analyzer and interactive viewer that runs in a terminal.<br>
&nbsp;&nbsp; <a href="https://github.com/lebinh/ngxtop"><b>ngxtop</b></a> - real-time metrics for nginx server.<br>
</p>
##### :black_small_square: Databases
<p>
&nbsp;&nbsp; <a href="https://github.com/xo/usql"><b>usql</b></a> - universal command-line interface for SQL databases.<br>
&nbsp;&nbsp; <a href="https://github.com/dbcli/pgcli"><b>pgcli</b></a> - postgres CLI with autocompletion and syntax highlighting.<br>
&nbsp;&nbsp; <a href="https://github.com/dbcli/mycli"><b>mycli</b></a> - terminal client for MySQL with autocompletion and syntax highlighting.<br>
&nbsp;&nbsp; <a href="https://github.com/dbcli/litecli"><b>litecli</b></a> - SQLite CLI with autocompletion and syntax highlighting.<br>
&nbsp;&nbsp; <a href="https://github.com/dbcli/mssql-cli"><b>mssql-cli</b></a> - SQL Server CLI with autocompletion and syntax highlighting.<br>
&nbsp;&nbsp; <a href="https://github.com/osquery/osquery"><b>OSQuery</b></a> - is a SQL powered operating system instrumentation, monitoring, and analytics framework.<br>
&nbsp;&nbsp; <a href="https://github.com/ankane/pgsync"><b>pgsync</b></a> - sync data from one Postgres database to another.<br>
&nbsp;&nbsp; <a href="https://github.com/laixintao/iredis"><b>iredis</b></a> - a terminal client for redis with autocompletion and syntax highlighting.<br>
&nbsp;&nbsp; <a href="https://www.schemacrawler.com/diagramming.html"><b>SchemaCrawler</b></a> - generates an E-R diagram of your database.<br>
</p>
##### :black_small_square: TOR
<p>
&nbsp;&nbsp; <a href="https://github.com/GouveaHeitor/nipe"><b>Nipe</b></a> - script to make Tor Network your default gateway.<br>
&nbsp;&nbsp; <a href="https://github.com/trimstray/multitor"><b>multitor</b></a> - a tool that lets you create multiple TOR instances with a load-balancing.<br>
</p>
##### :black_small_square: Messengers/IRC Clients
<p>
&nbsp;&nbsp; <a href="https://irssi.org"><b>Irssi</b></a> - is a free open source terminal based IRC client.<br>
&nbsp;&nbsp; <a href="https://weechat.org/"><b>WeeChat</b></a> - is an extremely extensible and lightweight IRC client.<br>
</p>
##### :black_small_square: Productivity
<p>
&nbsp;&nbsp; <a href="https://taskwarrior.org"><b>taskwarrior</b></a> - task management system, todo list <br>
</p>
##### :black_small_square: Other
<p>
&nbsp;&nbsp; <a href="https://github.com/skx/sysadmin-util"><b>sysadmin-util</b></a> - tools for Linux/Unix sysadmins.<br>
&nbsp;&nbsp; <a href="http://inotify.aiken.cz/"><b>incron</b></a> - is an inode-based filesystem notification technology.<br>
&nbsp;&nbsp; <a href="https://github.com/axkibe/lsyncd"><b>lsyncd</b></a> - synchronizes local directories with remote targets (Live Syncing Daemon).<br>
&nbsp;&nbsp; <a href="https://github.com/rgburke/grv"><b>GRV</b></a> - is a terminal based interface for viewing Git repositories.<br>
&nbsp;&nbsp; <a href="https://jonas.github.io/tig/"><b>Tig</b></a> - text-mode interface for Git.<br>
&nbsp;&nbsp; <a href="https://github.com/tldr-pages/tldr"><b>tldr</b></a> - simplified and community-driven man pages.<br>
&nbsp;&nbsp; <a href="https://github.com/mholt/archiver"><b>archiver</b></a> - easily create and extract .zip, .tar, .tar.gz, .tar.bz2, .tar.xz, .tar.lz4, .tar.sz, and .rar.<br>
&nbsp;&nbsp; <a href="https://github.com/tj/commander.js"><b>commander.js</b></a> - minimal CLI creator in JavaScript.<br>
&nbsp;&nbsp; <a href="https://github.com/tomnomnom/gron"><b>gron</b></a> - make JSON greppable!<br>
&nbsp;&nbsp; <a href="https://github.com/itchyny/bed"><b>bed</b></a> - binary editor written in Go.<br>
</p>
#### GUI Tools &nbsp;[<sup>[TOC]</sup>](#anger-table-of-contents)
@@ -0,0 +1,251 @@
#### Hacking/Penetration Testing &nbsp;[<sup>[TOC]</sup>](#anger-table-of-contents)
##### :black_small_square: Pentesters arsenal tools
<p>
&nbsp;&nbsp; <a href="http://www.syhunt.com/sandcat/"><b>Sandcat Browser</b></a> - a penetration-oriented browser with plenty of advanced functionality already built in.<br>
&nbsp;&nbsp; <a href="https://www.metasploit.com/"><b>Metasploit</b></a> - tool and framework for pentesting system, web and many more.<br>
&nbsp;&nbsp; <a href="https://portswigger.net/burp"><b>Burp Suite</b></a> - tool for testing web app security, intercepting proxy to replay, inject, scan and fuzz.<br>
&nbsp;&nbsp; <a href="https://www.owasp.org/index.php/OWASP_Zed_Attack_Proxy_Project"><b>OWASP Zed Attack Proxy</b></a> - intercepting proxy to replay, inject, scan and fuzz HTTP requests.<br>
&nbsp;&nbsp; <a href="http://w3af.org/"><b>w3af</b></a> - is a Web Application Attack and Audit Framework.<br>
&nbsp;&nbsp; <a href="https://mitmproxy.org/"><b>mitmproxy</b></a> - an interactive TLS-capable intercepting HTTP proxy for penetration testers.<br>
&nbsp;&nbsp; <a href="https://cirt.net/Nikto2"><b>Nikto2</b></a> - web server scanner which performs comprehensive tests against web servers for multiple items.<br>
&nbsp;&nbsp; <a href="http://sqlmap.org/"><b>sqlmap</b></a> - tool that automates the process of detecting and exploiting SQL injection flaws.<br>
&nbsp;&nbsp; <a href="https://github.com/lanmaster53/recon-ng"><b>Recon-ng</b></a> - is a full-featured Web Reconnaissance framework written in Python.<br>
&nbsp;&nbsp; <a href="https://github.com/Tib3rius/AutoRecon"><b>AutoRecon</b></a> - is a network reconnaissance tool which performs automated enumeration of services.<br>
&nbsp;&nbsp; <a href="https://www.faradaysec.com/"><b>Faraday</b></a> - an Integrated Multiuser Pentest Environment.<br>
&nbsp;&nbsp; <a href="https://github.com/s0md3v/Photon"><b>Photon</b></a> - incredibly fast crawler designed for OSINT.<br>
&nbsp;&nbsp; <a href="https://github.com/s0md3v/XSStrike"><b>XSStrike</b></a> - most advanced XSS detection suite.<br>
&nbsp;&nbsp; <a href="https://github.com/1N3/Sn1per"><b>Sn1per</b></a> - automated pentest framework for offensive security experts.<br>
&nbsp;&nbsp; <a href="https://github.com/future-architect/vuls"><b>vuls</b></a> - is an agent-less vulnerability scanner for Linux, FreeBSD, and other.<br>
&nbsp;&nbsp; <a href="https://github.com/google/tsunami-security-scanner"><b>tsunami</b></a> - is a general purpose network security scanner with an extensible plugin system.<br>
&nbsp;&nbsp; <a href="https://github.com/michenriksen/aquatone"><b>aquatone</b></a> - a tool for domain flyovers.<br>
&nbsp;&nbsp; <a href="https://github.com/GitHackTools/BillCipher"><b>BillCipher</b></a> - information gathering tool for a website or IP address.<br>
&nbsp;&nbsp; <a href="https://github.com/Ekultek/WhatWaf"><b>WhatWaf</b></a> - detect and bypass web application firewalls and protection systems.<br>
&nbsp;&nbsp; <a href="https://github.com/s0md3v/Corsy"><b>Corsy</b></a> - CORS misconfiguration scanner.<br>
&nbsp;&nbsp; <a href="https://github.com/evyatarmeged/Raccoon"><b>Raccoon</b></a> - is a high performance offensive security tool for reconnaissance and vulnerability scanning.<br>
&nbsp;&nbsp; <a href="https://github.com/Nekmo/dirhunt"><b>dirhunt</b></a> - find web directories without bruteforce.<br>
&nbsp;&nbsp; <a href="https://www.openwall.com/john/"><b>John The Ripper</b></a> - is a fast password cracker, currently available for many flavors of Unix, Windows, and other.<br>
&nbsp;&nbsp; <a href="https://hashcat.net/hashcat/"><b>hashcat</b></a> - world's fastest and most advanced password recovery utility.<br>
&nbsp;&nbsp; <a href="http://lcamtuf.coredump.cx/p0f3/"><b>p0f</b></a> - is a tool to identify the players behind any incidental TCP/IP communications.<br>
&nbsp;&nbsp; <a href="https://github.com/mozilla/ssh_scan"><b>ssh_scan</b></a> - a prototype SSH configuration and policy scanner.<br>
&nbsp;&nbsp; <a href="https://github.com/woj-ciech/LeakLooker"><b>LeakLooker</b></a> - find open databases - powered by Binaryedge.io<br>
&nbsp;&nbsp; <a href="https://github.com/offensive-security/exploitdb"><b>exploitdb</b></a> - searchable archive from The Exploit Database.<br>
&nbsp;&nbsp; <a href="https://github.com/vulnersCom/getsploit"><b>getsploit</b></a> - is a command line utility for searching and downloading exploits.<br>
&nbsp;&nbsp; <a href="https://github.com/zardus/ctf-tools"><b>ctf-tools</b></a> - some setup scripts for security research tools.<br>
&nbsp;&nbsp; <a href="https://github.com/Gallopsled/pwntools"><b>pwntools</b></a> - CTF framework and exploit development library.<br>
&nbsp;&nbsp; <a href="https://github.com/bl4de/security-tools"><b>security-tools</b></a> - collection of small security tools created mostly in Python. CTFs, pentests and so on.<br>
&nbsp;&nbsp; <a href="https://github.com/leonteale/pentestpackage"><b>pentestpackage</b></a> - is a package of Pentest scripts.<br>
&nbsp;&nbsp; <a href="https://github.com/dloss/python-pentest-tools"><b>python-pentest-tools</b></a> - python tools for penetration testers.<br>
&nbsp;&nbsp; <a href="https://github.com/fuzzdb-project/fuzzdb"><b>fuzzdb</b></a> - dictionary of attack patterns and primitives for black-box application fault injection.<br>
&nbsp;&nbsp; <a href="https://github.com/google/AFL"><b>AFL</b></a> - is a free software fuzzer maintained by Google.<br>
&nbsp;&nbsp; <a href="https://github.com/AFLplusplus/AFLplusplus"><b>AFL++</b></a> - is AFL with community patches.<br>
&nbsp;&nbsp; <a href="https://github.com/google/syzkaller"><b>syzkaller</b></a> - is an unsupervised, coverage-guided kernel fuzzer.<br>
&nbsp;&nbsp; <a href="https://github.com/pwndbg/pwndbg"><b>pwndbg</b></a> - exploit development and reverse engineering with GDB made easy.<br>
&nbsp;&nbsp; <a href="https://github.com/longld/peda"><b>GDB PEDA</b></a> - Python Exploit Development Assistance for GDB.<br>
&nbsp;&nbsp; <a href="https://www.hex-rays.com/products/ida/index.shtml"><b>IDA</b></a> - multi-processor disassembler and debugger useful for reverse engineering malware.<br>
&nbsp;&nbsp; <a href="https://github.com/radare/radare2"><b>radare2</b></a> - framework for reverse-engineering and analyzing binaries.<br>
&nbsp;&nbsp; <a href="https://github.com/threat9/routersploit"><b>routersploit</b></a> - exploitation framework for embedded devices.<br>
&nbsp;&nbsp; <a href="https://github.com/NationalSecurityAgency/ghidra"><b>Ghidra</b></a> - is a software reverse engineering (SRE) framework.<br>
&nbsp;&nbsp; <a href="https://cutter.re/"><b>Cutter</b></a> - is an SRE platform integrating Ghidra's decompiler.<br>
&nbsp;&nbsp; <a href="https://github.com/salesforce/vulnreport"><b>Vulnreport</b></a> - open-source pentesting management and automation platform by Salesforce Product Security.<br>
&nbsp;&nbsp; <a href="https://github.com/sc0tfree/mentalist"><b>Mentalist</b></a> - is a graphical tool for custom wordlist generation.<br>
&nbsp;&nbsp; <a href="https://github.com/archerysec/archerysec"><b>archerysec</b></a> - vulnerability assessment and management helps to perform scans and manage vulnerabilities.<br>
&nbsp;&nbsp; <a href="https://github.com/j3ssie/Osmedeus"><b>Osmedeus</b></a> - fully automated offensive security tool for reconnaissance and vulnerability scanning.<br>
&nbsp;&nbsp; <a href="https://github.com/beefproject/beef"><b>beef</b></a> - the browser exploitation framework project.<br>
&nbsp;&nbsp; <a href="https://github.com/NullArray/AutoSploit"><b>AutoSploit</b></a> - automated mass exploiter.<br>
&nbsp;&nbsp; <a href="https://github.com/TH3xACE/SUDO_KILLER"><b>SUDO_KILLER</b></a> - is a tool to identify and exploit sudo rules' misconfigurations and vulnerabilities.<br>
&nbsp;&nbsp; <a href="https://github.com/VirusTotal/yara"><b>yara</b></a> - the pattern matching swiss knife.<br>
&nbsp;&nbsp; <a href="https://github.com/gentilkiwi/mimikatz"><b>mimikatz</b></a> - a little tool to play with Windows security.<br>
&nbsp;&nbsp; <a href="https://github.com/sherlock-project/sherlock"><b>sherlock</b></a> - hunt down social media accounts by username across social networks.<br>
&nbsp;&nbsp; <a href="https://owasp.org/www-project-threat-dragon/"><b>OWASP Threat Dragon</b></a> - is a tool used to create threat model diagrams and to record possible threats.<br>
</p>
##### :black_small_square: Pentests bookmarks collection
<p>
&nbsp;&nbsp; <a href="http://www.pentest-standard.org/index.php/Main_Page"><b>PTES</b></a> - the penetration testing execution standard.<br>
&nbsp;&nbsp; <a href="https://www.amanhardikar.com/mindmaps/Practice.html"><b>Pentests MindMap</b></a> - amazing mind map with vulnerable apps and systems.<br>
&nbsp;&nbsp; <a href="https://www.amanhardikar.com/mindmaps/webapptest.html"><b>WebApps Security Tests MindMap</b></a> - incredible mind map for WebApps security tests.<br>
&nbsp;&nbsp; <a href="https://brutelogic.com.br/blog/"><b>Brute XSS</b></a> - master the art of Cross Site Scripting.<br>
&nbsp;&nbsp; <a href="https://portswigger.net/web-security/cross-site-scripting/cheat-sheet"><b>XSS cheat sheet</b></a> - contains many vectors that can help you bypass WAFs and filters.<br>
&nbsp;&nbsp; <a href="https://jivoi.github.io/2015/07/03/offensive-security-bookmarks/"><b>Offensive Security Bookmarks</b></a> - security bookmarks collection, all things that author need to pass OSCP.<br>
&nbsp;&nbsp; <a href="https://github.com/coreb1t/awesome-pentest-cheat-sheets"><b>Awesome Pentest Cheat Sheets</b></a> - collection of the cheat sheets useful for pentesting.<br>
&nbsp;&nbsp; <a href="https://github.com/Hack-with-Github/Awesome-Hacking"><b>Awesome Hacking by HackWithGithub</b></a> - awesome lists for hackers, pentesters and security researchers.<br>
&nbsp;&nbsp; <a href="https://github.com/carpedm20/awesome-hacking"><b>Awesome Hacking by carpedm20</b></a> - a curated list of awesome hacking tutorials, tools and resources.<br>
&nbsp;&nbsp; <a href="https://github.com/vitalysim/Awesome-Hacking-Resources"><b>Awesome Hacking Resources</b></a> - collection of hacking/penetration testing resources to make you better.<br>
&nbsp;&nbsp; <a href="https://github.com/enaqx/awesome-pentest"><b>Awesome Pentest</b></a> - collection of awesome penetration testing resources, tools and other shiny things.<br>
&nbsp;&nbsp; <a href="https://github.com/m4ll0k/Awesome-Hacking-Tools"><b>Awesome-Hacking-Tools</b></a> - is a curated list of awesome Hacking Tools.<br>
&nbsp;&nbsp; <a href="https://github.com/ksanchezcld/Hacking_Cheat_Sheet"><b>Hacking Cheat Sheet</b></a> - author hacking and pentesting notes.<br>
&nbsp;&nbsp; <a href="https://github.com/toolswatch/blackhat-arsenal-tools"><b>blackhat-arsenal-tools</b></a> - official Black Hat arsenal security tools repository.<br>
&nbsp;&nbsp; <a href="https://www.peerlyst.com/posts/the-complete-list-of-infosec-related-cheat-sheets-claus-cramon"><b>Penetration Testing and WebApp Cheat Sheets</b></a> - the complete list of Infosec related cheat sheets.<br>
&nbsp;&nbsp; <a href="https://github.com/The-Art-of-Hacking/h4cker"><b>Cyber Security Resources</b></a> - includes thousands of cybersecurity-related references and resources.<br>
&nbsp;&nbsp; <a href="https://github.com/jhaddix/pentest-bookmarks"><b>Pentest Bookmarks</b></a> - there are a LOT of pentesting blogs.<br>
&nbsp;&nbsp; <a href="https://github.com/OlivierLaflamme/Cheatsheet-God"><b>Cheatsheet-God</b></a> - Penetration Testing Reference Bank - OSCP/PTP & PTX Cheatsheet.<br>
&nbsp;&nbsp; <a href="https://github.com/Cyb3rWard0g/ThreatHunter-Playbook"><b>ThreatHunter-Playbook</b></a> - to aid the development of techniques and hypothesis for hunting campaigns.<br>
&nbsp;&nbsp; <a href="https://github.com/hmaverickadams/Beginner-Network-Pentesting"><b>Beginner-Network-Pentesting</b></a> - notes for beginner network pentesting course.<br>
&nbsp;&nbsp; <a href="https://github.com/rewardone/OSCPRepo"><b>OSCPRepo</b></a> - is a list of resources that author have been gathering in preparation for the OSCP.<br>
&nbsp;&nbsp; <a href="https://github.com/swisskyrepo/PayloadsAllTheThings"><b>PayloadsAllTheThings</b></a> - a list of useful payloads and bypass for Web Application Security and Pentest/CTF.<br>
&nbsp;&nbsp; <a href="https://github.com/foospidy/payloads"><b>payloads</b></a> - git all the Payloads! A collection of web attack payloads.<br>
&nbsp;&nbsp; <a href="https://github.com/payloadbox/command-injection-payload-list"><b>command-injection-payload-list</b></a> - command injection payload list.<br>
&nbsp;&nbsp; <a href="https://github.com/jakejarvis/awesome-shodan-queries"><b>Awesome Shodan Search Queries</b></a> - great search queries to plug into Shodan.<br>
&nbsp;&nbsp; <a href="https://github.com/s0md3v/AwesomeXSS"><b>AwesomeXSS</b></a> - is a collection of Awesome XSS resources.<br>
&nbsp;&nbsp; <a href="https://github.com/JohnTroony/php-webshells"><b>php-webshells</b></a> - common php webshells.<br>
&nbsp;&nbsp; <a href="https://highon.coffee/blog/penetration-testing-tools-cheat-sheet/"><b>Pentesting Tools Cheat Sheet</b></a> - a quick reference high level overview for typical penetration testing.<br>
&nbsp;&nbsp; <a href="https://cheatsheetseries.owasp.org/"><b>OWASP Cheat Sheet Series</b></a> - is a collection of high value information on specific application security topics.<br>
&nbsp;&nbsp; <a href="https://jeremylong.github.io/DependencyCheck/index.html"><b>OWASP dependency-check</b></a> - is an open source solution the OWASP Top 10 2013 entry.<br>
&nbsp;&nbsp; <a href="https://www.owasp.org/index.php/OWASP_Proactive_Controls"><b>OWASP ProActive Controls</b></a> - OWASP Top 10 Proactive Controls 2018.<br>
&nbsp;&nbsp; <a href="https://github.com/blaCCkHatHacEEkr/PENTESTING-BIBLE"><b>PENTESTING-BIBLE</b></a> - hacking & penetration testing & red team & cyber security resources.<br>
&nbsp;&nbsp; <a href="https://github.com/nixawk/pentest-wiki"><b>pentest-wiki</b></a> - is a free online security knowledge library for pentesters/researchers.<br>
&nbsp;&nbsp; <a href="https://media.defcon.org/"><b>DEF CON Media Server</b></a> - great stuff from DEFCON.<br>
&nbsp;&nbsp; <a href="https://github.com/rshipp/awesome-malware-analysis"><b>Awesome Malware Analysis</b></a> - a curated list of awesome malware analysis tools and resources.<br>
&nbsp;&nbsp; <a href="https://www.netsparker.com/blog/web-security/sql-injection-cheat-sheet/"><b>SQL Injection Cheat Sheet</b></a> - detailed technical stuff about the many different variants of the SQL Injection.<br>
&nbsp;&nbsp; <a href="http://kb.entersoft.co.in/"><b>Entersoft Knowledge Base</b></a> - great and detailed reference about vulnerabilities.<br>
&nbsp;&nbsp; <a href="http://html5sec.org/"><b>HTML5 Security Cheatsheet</b></a> - a collection of HTML5 related XSS attack vectors.<br>
&nbsp;&nbsp; <a href="http://evuln.com/tools/xss-encoder/"><b>XSS String Encoder</b></a> - for generating XSS code to check your input validation filters against XSS.<br>
&nbsp;&nbsp; <a href="https://gtfobins.github.io/"><b>GTFOBins</b></a> - list of Unix binaries that can be exploited by an attacker to bypass local security restrictions.<br>
&nbsp;&nbsp; <a href="https://guif.re/"><b>Guifre Ruiz Notes</b></a> - collection of security, system, network and pentest cheatsheets.<br>
&nbsp;&nbsp; <a href="http://blog.safebuff.com/2016/07/03/SSRF-Tips/index.html"><b>SSRF Tips</b></a> - a collection of SSRF Tips.<br>
&nbsp;&nbsp; <a href="http://shell-storm.org/repo/CTF/"><b>shell-storm repo CTF</b></a> - great archive of CTFs.<br>
&nbsp;&nbsp; <a href="https://github.com/bl4de/ctf"><b>ctf</b></a> - CTF (Capture The Flag) writeups, code snippets, notes, scripts.<br>
&nbsp;&nbsp; <a href="https://github.com/orangetw/My-CTF-Web-Challenges"><b>My-CTF-Web-Challenges</b></a> - collection of CTF Web challenges.<br>
&nbsp;&nbsp; <a href="https://github.com/OWASP/owasp-mstg"><b>MSTG</b></a> - The Mobile Security Testing Guide (MSTG) is a comprehensive manual for mobile app security testing.<br>
&nbsp;&nbsp; <a href="https://github.com/sdcampbell/Internal-Pentest-Playbook"><b>Internal-Pentest-Playbook</b></a> - notes on the most common things for an Internal Network Penetration Test.<br>
&nbsp;&nbsp; <a href="https://github.com/streaak/keyhacks"><b>KeyHacks</b></a> - shows quick ways in which API keys leaked by a bug bounty program can be checked.<br>
&nbsp;&nbsp; <a href="https://github.com/securitum/research"><b>securitum/research</b></a> - various Proof of Concepts of security research performed by Securitum.<br>
&nbsp;&nbsp; <a href="https://github.com/juliocesarfort/public-pentesting-reports"><b>public-pentesting-reports</b></a> - is a list of public pentest reports released by several consulting security groups.<br>
&nbsp;&nbsp; <a href="https://github.com/djadmin/awesome-bug-bounty"><b>awesome-bug-bounty</b></a> - is a comprehensive curated list of available Bug Bounty.<br>
&nbsp;&nbsp; <a href="https://github.com/ngalongc/bug-bounty-reference"><b>bug-bounty-reference</b></a> - is a list of bug bounty write-ups.<br>
&nbsp;&nbsp; <a href="https://github.com/devanshbatham/Awesome-Bugbounty-Writeups"><b>Awesome-Bugbounty-Writeups</b></a> - is a curated list of bugbounty writeups.<br>
&nbsp;&nbsp; <a href="https://pentester.land/list-of-bug-bounty-writeups.html"><b>Bug bounty writeups</b></a> - list of bug bounty writeups (2012-2020).<br>
&nbsp;&nbsp; <a href="https://hackso.me/"><b>hackso.me</b></a> - a great journey into security.<br>
</p>
##### :black_small_square: Backdoors/exploits
<p>
&nbsp;&nbsp; <a href="https://github.com/bartblaze/PHP-backdoors"><b>PHP-backdoors</b></a> - a collection of PHP backdoors. For educational or testing purposes only.<br>
</p>
##### :black_small_square: Wordlists and Weak passwords
<p>
&nbsp;&nbsp; <a href="https://weakpass.com/"><b>Weakpass</b></a> - for any kind of bruteforce find wordlists or unleash the power of them all at once!<br>
&nbsp;&nbsp; <a href="https://hashes.org/"><b>Hashes.org</b></a> - is a free online hash resolving service incorporating many unparalleled techniques.<br>
&nbsp;&nbsp; <a href="https://github.com/danielmiessler/SecLists"><b>SecLists</b></a> - collection of multiple types of lists used during security assessments, collected in one place.<br>
&nbsp;&nbsp; <a href="https://github.com/berzerk0/Probable-Wordlists"><b>Probable-Wordlists</b></a> - sorted by probability originally created for password generation and testing.<br>
&nbsp;&nbsp; <a href="https://wiki.skullsecurity.org/index.php?title=Passwords"><b>skullsecurity passwords</b></a> - password dictionaries and leaked passwords repository.<br>
&nbsp;&nbsp; <a href="https://bezpieka.org/polski-slownik-premium-polish-wordlist"><b>Polish PREMIUM Dictionary</b></a> - official dictionary created by the team on the forum bezpieka.org.<b>*</b> <sup><a href="https://sourceforge.net/projects/kali-linux/files/Wordlist/">1</sup><br>
&nbsp;&nbsp; <a href="https://github.com/insidetrust/statistically-likely-usernames"><b>statistically-likely-usernames</b></a> - wordlists for creating statistically likely username lists.<br>
</p>
##### :black_small_square: Bounty platforms
<p>
&nbsp;&nbsp; <a href="https://www.yeswehack.com/"><b>YesWeHack</b></a> - bug bounty platform with infosec jobs.<br>
&nbsp;&nbsp; <a href="https://www.openbugbounty.org/"><b>Openbugbounty</b></a> - allows any security researcher reporting a vulnerability on any website.<br>
&nbsp;&nbsp; <a href="https://www.hackerone.com/"><b>hackerone</b></a> - global hacker community to surface the most relevant security issues.<br>
&nbsp;&nbsp; <a href="https://www.bugcrowd.com/"><b>bugcrowd</b></a> - crowdsourced cybersecurity for the enterprise.<br>
&nbsp;&nbsp; <a href="https://crowdshield.com/"><b>Crowdshield</b></a> - crowdsourced security & bug bounty management.<br>
&nbsp;&nbsp; <a href="https://www.synack.com/"><b>Synack</b></a> - crowdsourced security & bug bounty programs, crowd security intelligence platform, and more.<br>
&nbsp;&nbsp; <a href="https://hacktrophy.com/en/"><b>Hacktrophy</b></a> - bug bounty platform.<br>
</p>
##### :black_small_square: Web Training Apps (local installation)
<p>
&nbsp;&nbsp; <a href="https://www.owasp.org/index.php/OWASP_Vulnerable_Web_Applications_Directory_Project"><b>OWASP-VWAD</b></a> - comprehensive and well maintained registry of all known vulnerable web applications.<br>
&nbsp;&nbsp; <a href="http://www.dvwa.co.uk/"><b>DVWA</b></a> - PHP/MySQL web application that is damn vulnerable.<br>
&nbsp;&nbsp; <a href="https://metasploit.help.rapid7.com/docs/metasploitable-2"><b>metasploitable2</b></a> - vulnerable web application amongst security researchers.<br>
&nbsp;&nbsp; <a href="https://github.com/rapid7/metasploitable3"><b>metasploitable3</b></a> - is a VM that is built from the ground up with a large amount of security vulnerabilities.<br>
&nbsp;&nbsp; <a href="https://github.com/stamparm/DSVW"><b>DSVW</b></a> - is a deliberately vulnerable web application written in under 100 lines of code.<br>
&nbsp;&nbsp; <a href="https://sourceforge.net/projects/mutillidae/"><b>OWASP Mutillidae II</b></a> - free, open source, deliberately vulnerable web-application.<br>
&nbsp;&nbsp; <a href="https://www.owasp.org/index.php/OWASP_Juice_Shop_Project"><b>OWASP Juice Shop Project</b></a> - the most bug-free vulnerable application in existence.<br>
&nbsp;&nbsp; <a href="https://www.owasp.org/index.php/Projects/OWASP_Node_js_Goat_Project"><b>OWASP Node js Goat Project</b></a> - OWASP Top 10 security risks apply to web apps developed using Node.js.<br>
&nbsp;&nbsp; <a href="https://github.com/iteratec/juicy-ctf"><b>juicy-ctf</b></a> - run Capture the Flags and Security Trainings with OWASP Juice Shop.<br>
&nbsp;&nbsp; <a href="https://github.com/OWASP/SecurityShepherd"><b>SecurityShepherd</b></a> - web and mobile application security training platform.<br>
&nbsp;&nbsp; <a href="https://github.com/opendns/Security_Ninjas_AppSec_Training"><b>Security Ninjas</b></a> - open source application security training program.<br>
&nbsp;&nbsp; <a href="https://github.com/rapid7/hackazon"><b>hackazon</b></a> - a modern vulnerable web app.<br>
&nbsp;&nbsp; <a href="https://github.com/appsecco/dvna"><b>dvna</b></a> - damn vulnerable NodeJS application.<br>
&nbsp;&nbsp; <a href="https://github.com/DefectDojo/django-DefectDojo"><b>django-DefectDojo</b></a> - is an open-source application vulnerability correlation and security orchestration tool.<br>
&nbsp;&nbsp; <a href="https://google-gruyere.appspot.com/"><b>Google Gruyere</b></a> - web application exploits and defenses.<br>
&nbsp;&nbsp; <a href="https://github.com/amolnaik4/bodhi"><b>Bodhi</b></a> - is a playground focused on learning the exploitation of client-side web vulnerabilities.<br>
&nbsp;&nbsp; <a href="https://websploit.h4cker.org/"><b>Websploit</b></a> - single vm lab with the purpose of combining several vulnerable appliations in one environment.<br>
&nbsp;&nbsp; <a href="https://github.com/vulhub/vulhub"><b>vulhub</b></a> - pre-built Vulnerable Environments based on docker-compose.<br>
&nbsp;&nbsp; <a href="https://rhinosecuritylabs.com/aws/introducing-cloudgoat-2/"><b>CloudGoat 2</b></a> - the new & improved "Vulnerable by Design"
AWS deployment tool.<br>
&nbsp;&nbsp; <a href="https://github.com/globocom/secDevLabs"><b>secDevLabs</b></a> - is a laboratory for learning secure web development in a practical manner.<br>
&nbsp;&nbsp; <a href="https://github.com/incredibleindishell/CORS-vulnerable-Lab"><b>CORS-vulnerable-Lab</b></a> - sample vulnerable code and its exploit code.<br>
&nbsp;&nbsp; <a href="https://github.com/moloch--/RootTheBox"><b>RootTheBox</b></a> - a Game of Hackers (CTF Scoreboard & Game Manager).<br>
&nbsp;&nbsp; <a href="https://application.security/"><b>KONTRA</b></a> - application security training (OWASP Top Web & Api).<br>
</p>
##### :black_small_square: Labs (ethical hacking platforms/trainings/CTFs)
<p>
&nbsp;&nbsp; <a href="https://www.offensive-security.com/"><b>Offensive Security</b></a> - true performance-based penetration testing training for over a decade.<br>
&nbsp;&nbsp; <a href="https://www.hackthebox.eu/"><b>Hack The Box</b></a> - online platform allowing you to test your penetration testing skills.<br>
&nbsp;&nbsp; <a href="https://www.hacking-lab.com/index.html"><b>Hacking-Lab</b></a> - online ethical hacking, computer network and security challenge platform.<br>
&nbsp;&nbsp; <a href="http://pwnable.kr/index.php"><b>pwnable.kr</b></a> - non-commercial wargame site which provides various pwn challenges.<br>
&nbsp;&nbsp; <a href="https://pwnable.tw/"><b>Pwnable.tw</b></a> - is a wargame site for hackers to test and expand their binary exploiting skills.<br>
&nbsp;&nbsp; <a href="https://picoctf.com/"><b>picoCTF</b></a> - is a free computer security game targeted at middle and high school students.<br>
&nbsp;&nbsp; <a href="https://ctflearn.com/"><b>CTFlearn</b></a> - is an online platform built to help ethical hackers learn and practice their cybersecurity knowledge.<br>
&nbsp;&nbsp; <a href="https://ctftime.org/"><b>ctftime</b></a> - CTF archive and a place, where you can get some another CTF-related info.<br>
&nbsp;&nbsp; <a href="https://silesiasecuritylab.com/"><b>Silesia Security Lab</b></a> - high quality security testing services.<br>
&nbsp;&nbsp; <a href="https://practicalpentestlabs.com/"><b>Practical Pentest Labs</b></a> - pentest lab, take your Hacking skills to the next level.<br>
&nbsp;&nbsp; <a href="https://www.root-me.org/?lang=en"><b>Root Me</b></a> - the fast, easy, and affordable way to train your hacking skills.<br>
&nbsp;&nbsp; <a href="https://rozwal.to/login"><b>rozwal.to</b></a> - a great platform to train your pentesting skills.<br>
&nbsp;&nbsp; <a href="https://tryhackme.com/"><b>TryHackMe</b></a> - learning Cyber Security made easy.<br>
&nbsp;&nbsp; <a href="https://hackxor.net/"><b>hackxor</b></a> - is a realistic web application hacking game, designed to help players of all abilities develop their skills.<br>
&nbsp;&nbsp; <a href="http://hack-yourself-first.com/"><b>Hack Yourself First</b></a> - it's full of nasty app sec holes.<br>
&nbsp;&nbsp; <a href="http://overthewire.org/wargames/"><b>OverTheWire</b></a> - can help you to learn and practice security concepts in the form of fun-filled games.<br>
&nbsp;&nbsp; <a href="https://labs.wizard-security.net/"><b>Wizard Labs</b></a> - is an online Penetration Testing Lab.<br>
&nbsp;&nbsp; <a href="https://pentesterlab.com/"><b>PentesterLab</b></a> - provides vulnerable systems that can be used to test and understand vulnerabilities.<br>
&nbsp;&nbsp; <a href="https://ringzer0ctf.com/"><b>RingZer0</b></a> - tons of challenges designed to test and improve your hacking skills.<br>
&nbsp;&nbsp; <a href="http://www.try2hack.nl/"><b>try2hack</b></a> - several security-oriented challenges for your entertainment.<br>
&nbsp;&nbsp; <a href="https://www.ubeeri.com/preconfig-labs"><b>Ubeeri</b></a> - preconfigured lab environments.<br>
&nbsp;&nbsp; <a href="https://lab.pentestit.ru/"><b>Pentestit</b></a> - emulate IT infrastructures of real companies for legal pen testing and improving pentest skills.<br>
&nbsp;&nbsp; <a href="https://microcorruption.com/login"><b>Microcorruption</b></a> - reversal challenges done in the web interface.<br>
&nbsp;&nbsp; <a href="https://crackmes.one/"><b>Crackmes</b></a> - download crackmes to help improve your reverse engineering skills.<br>
&nbsp;&nbsp; <a href="https://domgo.at/cxss/intro"><b>DomGoat</b></a> - DOM XSS security learning and practicing platform.<br>
&nbsp;&nbsp; <a href="https://chall.stypr.com"><b>Stereotyped Challenges</b></a> - upgrade your web hacking techniques today!<br>
&nbsp;&nbsp; <a href="https://www.vulnhub.com/"><b>Vulnhub</b></a> - allows anyone to gain practical 'hands-on' experience in digital security.<br>
&nbsp;&nbsp; <a href="https://w3challs.com/"><b>W3Challs</b></a> - is a penetration testing training platform, which offers various computer challenges.<br>
&nbsp;&nbsp; <a href="https://ringzer0ctf.com/challenges"><b>RingZer0 CTF</b></a> - offers you tons of challenges designed to test and improve your hacking skills.<br>
&nbsp;&nbsp; <a href="https://hack.me/"><b>Hack.me</b></a> - a platform where you can build, host and share vulnerable web apps for educational purposes.<br>
&nbsp;&nbsp; <a href="https://www.hackthis.co.uk/levels/"><b>HackThis!</b></a> - discover how hacks, dumps and defacements are performed and secure your website.<br>
&nbsp;&nbsp; <a href="https://www.enigmagroup.org/#"><b>Enigma Group WebApp Training</b></a> - these challenges cover the exploits listed in the OWASP Top 10 Project.<br>
&nbsp;&nbsp; <a href="https://challenges.re/"><b>Reverse Engineering Challenges</b></a> - challenges, exercises, problems and tasks - by level, by type, and more.<br>
&nbsp;&nbsp; <a href="https://0x00sec.org/"><b>0x00sec</b></a> - the home of the Hacker - Malware, Reverse Engineering, and Computer Science.<br>
&nbsp;&nbsp; <a href="https://www.wechall.net/challs"><b>We Chall</b></a> - there are exist a lots of different challenge types.<br>
&nbsp;&nbsp; <a href="https://www.hackergateway.com/"><b>Hacker Gateway</b></a> - is the go-to place for hackers who want to test their skills.<br>
&nbsp;&nbsp; <a href="https://www.hacker101.com/"><b>Hacker101</b></a> - is a free class for web security.<br>
&nbsp;&nbsp; <a href="https://contained.af/"><b>contained.af</b></a> - a stupid game for learning about containers, capabilities, and syscalls.<br>
&nbsp;&nbsp; <a href="http://flaws.cloud/"><b>flAWS challenge!</b></a> - a series of levels you'll learn about common mistakes and gotchas when using AWS.<br>
&nbsp;&nbsp; <a href="https://cybersecurity.wtf"><b>CyberSec WTF</b></a> - provides web hacking challenges derived from bounty write-ups.<br>
&nbsp;&nbsp; <a href="https://ctfchallenge.co.uk/login"><b>CTF Challenge</b></a> - CTF Web App challenges.<br>
&nbsp;&nbsp; <a href="https://capturetheflag.withgoogle.com"><b>gCTF</b></a> - most of the challenges used in the Google CTF 2017.<br>
&nbsp;&nbsp; <a href="https://www.hackthissite.org/pages/index/index.php"><b>Hack This Site</b></a> - is a free, safe and legal training ground for hackers.<br>
&nbsp;&nbsp; <a href="https://attackdefense.com"><b>Attack & Defense</b></a> - is a browser-based cloud labs.<br>
&nbsp;&nbsp; <a href="https://cryptohack.org/"><b>Cryptohack</b></a> - a fun platform for learning modern cryptography.<br>
&nbsp;&nbsp; <a href="https://cryptopals.com/"><b>Cryptopals</b></a> - the cryptopals crypto challenges.<br>
</p>
##### :black_small_square: CTF platforms
<p>
&nbsp;&nbsp; <a href="https://github.com/facebook/fbctf"><b>fbctf</b></a> - platform to host Capture the Flag competitions.<br>
&nbsp;&nbsp; <a href="https://github.com/google/ctfscoreboard"><b>ctfscoreboard</b></a> - scoreboard for Capture The Flag competitions.<br>
</p>
##### :black_small_square: Other resources
<p>
&nbsp;&nbsp; <a href="https://github.com/bugcrowd/bugcrowd_university"><b>Bugcrowd University</b></a> - open source education content for the researcher community.<br>
&nbsp;&nbsp; <a href="https://github.com/rewardone/OSCPRepo"><b>OSCPRepo</b></a> - a list of resources and scripts that I have been gathering in preparation for the OSCP.<br>
&nbsp;&nbsp; <a href="https://medium.com/@cxosmo/owasp-top-10-real-world-examples-part-1-a540c4ea2df5"><b>OWASP Top 10: Real-World Examples</b></a> - test your web apps with real-world examples (two-part series).<br>
&nbsp;&nbsp; <a href="http://phrack.org/index.html"><b>phrack.org</b></a> - an awesome collection of articles from several respected hackers and other thinkers.<br>
&nbsp;&nbsp; <a href="https://github.com/Gr1mmie/Practical-Ethical-Hacking-Resources"><b>Practical-Ethical-Hacking-Resources</b></a> - compilation of resources from TCM's Udemy Course.<br>
</p>
#### Your daily knowledge and news &nbsp;[<sup>[TOC]</sup>](#anger-table-of-contents)
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,97 @@
#### Shell Tricks &nbsp;[<sup>[TOC]</sup>](#anger-table-of-contents)
When you get a shell, it is generally not very clean, but after following these steps, you will have a fairly clean and comfortable shell to work with.
1) `script /dev/null -c bash`
2) Ctrl-Z (to send it to background)
3) `stty raw -echo; fg` (returns the shell to foreground)
4) `reset` (to reset terminal)
5) `xterm` (when asked for terminal type)
6) `export TERM=xterm; export SHELL=bash`
#### Shell functions &nbsp;[<sup>[TOC]</sup>](#anger-table-of-contents)
##### Table of Contents
- [Domain resolve](#domain-resolve)
- [Get ASN](#get-asn)
###### Domain resolve
```bash
# Dependencies:
# - curl
# - jq
function DomainResolve() {
local _host="$1"
local _curl_base="curl --request GET"
local _timeout="15"
_host_ip=$($_curl_base -ks -m "$_timeout" "https://dns.google.com/resolve?name=${_host}&type=A" | \
jq '.Answer[0].data' | tr -d "\"" 2>/dev/null)
if [[ -z "$_host_ip" ]] || [[ "$_host_ip" == "null" ]] ; then
echo -en "Unsuccessful domain name resolution.\\n"
else
echo -en "$_host > $_host_ip\\n"
fi
}
```
Example:
```bash
shell> DomainResolve nmap.org
nmap.org > 45.33.49.119
shell> DomainResolve nmap.org
Unsuccessful domain name resolution.
```
###### Get ASN
```bash
# Dependencies:
# - curl
function GetASN() {
local _ip="$1"
local _curl_base="curl --request GET"
local _timeout="15"
_asn=$($_curl_base -ks -m "$_timeout" "http://ip-api.com/line/${_ip}?fields=as")
_state=$(echo $?)
if [[ -z "$_ip" ]] || [[ "$_ip" == "null" ]] || [[ "$_state" -ne 0 ]]; then
echo -en "Unsuccessful ASN gathering.\\n"
else
echo -en "$_ip > $_asn\\n"
fi
}
```
Example:
```bash
shell> GetASN 1.1.1.1
1.1.1.1 > AS13335 Cloudflare, Inc.
shell> GetASN 0.0.0.0
Unsuccessful ASN gathering.
```
@@ -0,0 +1,269 @@
#### Web Tools &nbsp;[<sup>[TOC]</sup>](#anger-table-of-contents)
##### :black_small_square: Browsers
<p>
&nbsp;&nbsp; <a href="https://www.ssllabs.com/ssltest/viewMyClient.html"><b>SSL/TLS Capabilities of Your Browser</b></a> - test your browser's SSL implementation.<br>
&nbsp;&nbsp; <a href="https://caniuse.com/"><b>Can I use</b></a> - provides up-to-date browser support tables for support of front-end web technologies.<br>
&nbsp;&nbsp; <a href="https://panopticlick.eff.org/"><b>Panopticlick 3.0</b></a> - is your browser safe against tracking?<br>
&nbsp;&nbsp; <a href="https://privacy.net/analyzer/"><b>Privacy Analyzer</b></a> - see what data is exposed from your browser.<br>
&nbsp;&nbsp; <a href="https://browserleaks.com/"><b>Web Browser Security</b></a> - it's all about Web Browser fingerprinting.<br>
&nbsp;&nbsp; <a href="https://www.howsmyssl.com/"><b>How's My SSL?</b></a> - help a web server developer learn what real world TLS clients were capable of.<br>
&nbsp;&nbsp; <a href="https://suche.org/sslClientInfo"><b>sslClientInfo</b></a> - client test (incl TLSv1.3 information).<br>
</p>
##### :black_small_square: SSL/Security
<p>
&nbsp;&nbsp; <a href="https://www.ssllabs.com/ssltest/"><b>SSLLabs Server Test</b></a> - performs a deep analysis of the configuration of any SSL web server.<br>
&nbsp;&nbsp; <a href="https://dev.ssllabs.com/ssltest/"><b>SSLLabs Server Test (DEV)</b></a> - performs a deep analysis of the configuration of any SSL web server.<br>
&nbsp;&nbsp; <a href="https://www.immuniweb.com/ssl/"><b>ImmuniWeb® SSLScan</b></a> - test SSL/TLS (PCI DSS, HIPAA and NIST).<br>
&nbsp;&nbsp; <a href="https://www.jitbit.com/sslcheck/"><b>SSL Check</b></a> - scan your website for non-secure content.<br>
&nbsp;&nbsp; <a href="http://www.ssltools.com"><b>SSL Scanner</b></a> - analyze website security.<br>
&nbsp;&nbsp; <a href="https://cryptcheck.fr/"><b>CryptCheck</b></a> - test your TLS server configuration (e.g. ciphers).<br>
&nbsp;&nbsp; <a href="https://urlscan.io/"><b>urlscan.io</b></a> - service to scan and analyse websites.<br>
&nbsp;&nbsp; <a href="https://report-uri.com/home/tools"><b>Report URI</b></a> - monitoring security policies like CSP and HPKP.<br>
&nbsp;&nbsp; <a href="https://csp-evaluator.withgoogle.com/"><b>CSP Evaluator</b></a> - allows developers and security experts to check if a Content Security Policy.<br>
&nbsp;&nbsp; <a href="https://uselesscsp.com/"><b>Useless CSP</b></a> - public list about CSP in some big players (might make them care a bit more).<br>
&nbsp;&nbsp; <a href="https://whynohttps.com/"><b>Why No HTTPS?</b></a> - top 100 websites by Alexa rank not automatically redirecting insecure requests.<br>
&nbsp;&nbsp; <a href="https://ciphersuite.info/"><b>TLS Cipher Suite Search</b></a>- cipher suite search engine.<br>
&nbsp;&nbsp; <a href="https://github.com/RaymiiOrg/cipherli.st"><b>cipherli.st</b></a> - strong ciphers for Apache, Nginx, Lighttpd, and more.<b>*</b><br>
&nbsp;&nbsp; <a href="https://2ton.com.au/dhtool/"><b>dhtool</b></a> - public Diffie-Hellman parameter service/tool.<br>
&nbsp;&nbsp; <a href="https://badssl.com/"><b>badssl.com</b></a> - memorable site for testing clients against bad SSL configs.<br>
&nbsp;&nbsp; <a href="https://tlsfun.de/"><b>tlsfun.de</b></a> - registered for various tests regarding the TLS/SSL protocol.<br>
&nbsp;&nbsp; <a href="https://sslmate.com/caa/"><b>CAA Record Helper</b></a> - generate a CAA policy.<br>
&nbsp;&nbsp; <a href="https://ccadb.org/resources"><b>Common CA Database</b></a> - repository of information about CAs, and their root and intermediate certificates.<br>
&nbsp;&nbsp; <a href="https://certstream.calidog.io/"><b>CERTSTREAM</b></a> - real-time certificate transparency log update stream.<br>
&nbsp;&nbsp; <a href="https://crt.sh/"><b>crt.sh</b></a> - discovers certificates by continually monitoring all of the publicly known CT.<br>
&nbsp;&nbsp; <a href="https://www.hardenize.com/"><b>Hardenize</b></a> - deploy the security standards.<br>
&nbsp;&nbsp; <a href="https://cryptcheck.fr/suite/"><b>Cipher suite compatibility</b></a> - test TLS cipher suite compatibility.<br>
&nbsp;&nbsp; <a href="https://www.urlvoid.com/"><b>urlvoid</b></a> - this service helps you detect potentially malicious websites.<br>
&nbsp;&nbsp; <a href="https://securitytxt.org/"><b>security.txt</b></a> - a proposed standard (generator) which allows websites to define security policies.<br>
&nbsp;&nbsp; <a href="https://github.com/mozilla/ssl-config-generator"><b>ssl-config-generator</b></a> - help you follow the Mozilla Server Side TLS configuration guidelines.<br>
&nbsp;&nbsp; <a href="https://github.com/mjol/TLScan"><b>TLScan</b></a> - pure python, SSL/TLS protocol and cipher scanner/enumerator.<br>
</p>
##### :black_small_square: HTTP Headers & Web Linters
<p>
&nbsp;&nbsp; <a href="https://securityheaders.com/"><b>Security Headers</b></a> - analyse the HTTP response headers (with rating system to the results).<br>
&nbsp;&nbsp; <a href="https://observatory.mozilla.org/"><b>Observatory by Mozilla</b></a> - set of tools to analyze your website.<br>
&nbsp;&nbsp; <a href="https://webhint.io/"><b>webhint</b></a> - is a linting tool that will help you with your site's accessibility, speed, security, and more.<br>
</p>
##### :black_small_square: DNS
<p>
&nbsp;&nbsp; <a href="http://viewdns.info/"><b>ViewDNS</b></a> - one source for free DNS related tools and information.<br>
&nbsp;&nbsp; <a href="https://dnslookup.org/"><b>DNSLookup</b></a> - is an advanced DNS lookup tool.<br>
&nbsp;&nbsp; <a href="https://dnslytics.com/"><b>DNSlytics</b></a> - online DNS investigation tool.<br>
&nbsp;&nbsp; <a href="https://dnsspy.io/"><b>DNS Spy</b></a> - monitor, validate and verify your DNS configurations.<br>
&nbsp;&nbsp; <a href="https://zonemaster.iis.se/en/"><b>Zonemaster</b></a> - helps you to control how your DNS works.<br>
&nbsp;&nbsp; <a href="http://leafdns.com/"><b>Leaf DNS</b></a> - comprehensive DNS tester.<br>
&nbsp;&nbsp; <a href="https://findsubdomains.com/"><b>Find subdomains online</b></a> - find subdomains for security assessment penetration test.<br>
&nbsp;&nbsp; <a href="https://dnsdumpster.com/"><b>DNSdumpster</b></a> - dns recon & research, find & lookup dns records.<br>
&nbsp;&nbsp; <a href="https://dnstable.com/"><b>DNS Table online</b></a> - search for DNS records by domain, IP, CIDR, ISP.<br>
&nbsp;&nbsp; <a href="https://intodns.com/"><b>intoDNS</b></a> - DNS and mail server health checker.<br>
&nbsp;&nbsp; <a href="http://www.zonecut.net/dns/"><b>DNS Bajaj</b></a> - check the delegation of your domain.<br>
&nbsp;&nbsp; <a href="https://www.buddyns.com/delegation-lab/"><b>BuddyDNS Delegation LAB</b></a> - check, trace and visualize delegation of your domain.<br>
&nbsp;&nbsp; <a href="https://dnssec-debugger.verisignlabs.com/"><b>dnssec-debugger</b></a> - DS or DNSKEY records validator.<br>
&nbsp;&nbsp; <a href="http://ptrarchive.com/"><b>PTRarchive.com</b></a> - this site is responsible for the safekeeping of historical reverse DNS records.<br>
&nbsp;&nbsp; <a href="http://xip.io/"><b>xip.io</b></a> - wildcard DNS for everyone.<br>
&nbsp;&nbsp; <a href="https://nip.io/"><b>nip.io</b></a> - dead simple wildcard DNS for any IP Address.<br>
&nbsp;&nbsp; <a href="https://ceipam.eu/en/dnslookup.php"><b>dnslookup (ceipam)</b></a> - one of the best DNS propagation checker (and not only).<br>
&nbsp;&nbsp; <a href="https://whatsmydns.com"><b>What's My DNS</b></a> - DNS propagation checking tool.<br>
&nbsp;&nbsp; <a href="https://blog.erbbysam.com/index.php/2019/02/09/dnsgrep/"><b>DNSGrep</b></a> - quickly searching large DNS datasets.<br>
</p>
##### :black_small_square: Mail
<p>
&nbsp;&nbsp; <a href="https://luxsci.com/smtp-tls-checker"><b>smtp-tls-checker</b></a> - check an email domain for SMTP TLS support.<br>
&nbsp;&nbsp; <a href="https://mxtoolbox.com/SuperTool.aspx"><b>MX Toolbox</b></a> - all of your MX record, DNS, blacklist and SMTP diagnostics in one integrated tool.<br>
&nbsp;&nbsp; <a href="https://www.checktls.com/index.html"><b>Secure Email</b></a> - complete email test tools for email technicians.<br>
&nbsp;&nbsp; <a href="http://www.blacklistalert.org/"><b>blacklistalert</b></a> - checks to see if your domain is on a Real Time Spam Blacklist.<br>
&nbsp;&nbsp; <a href="http://multirbl.valli.org/"><b>MultiRBL</b></a> - complete IP check for sending Mailservers.<br>
&nbsp;&nbsp; <a href="https://dkimvalidator.com/"><b>DKIM SPF & Spam Assassin Validator</b></a> - checks mail authentication and scores messages with Spam Assassin.<br>
</p>
##### :black_small_square: Encoders/Decoders and Regex testing
<p>
&nbsp;&nbsp; <a href="https://www.url-encode-decode.com/"><b>URL Encode/Decode</b></a> - tool from above to either encode or decode a string of text.<br>
&nbsp;&nbsp; <a href="https://uncoder.io/"><b>Uncoder</b></a> - the online translator for search queries on log data.<br>
&nbsp;&nbsp; <a href="https://regex101.com/"><b>Regex101</b></a> - online regex tester and debugger: PHP, PCRE, Python, Golang and JavaScript.<br>
&nbsp;&nbsp; <a href="https://regexr.com/"><b>RegExr</b></a> - online tool to learn, build, & test Regular Expressions (RegEx / RegExp).<br>
&nbsp;&nbsp; <a href="https://www.regextester.com/"><b>RegEx Testing</b></a> - online regex testing tool.<br>
&nbsp;&nbsp; <a href="https://www.regexpal.com/"><b>RegEx Pal</b></a> - online regex testing tool + other tools.<br>
&nbsp;&nbsp; <a href="https://gchq.github.io/CyberChef/"><b>The Cyber Swiss Army Knife</b></a> - a web app for encryption, encoding, compression and data analysis.<br>
</p>
##### :black_small_square: Net-tools
<p>
&nbsp;&nbsp; <a href="https://toolbar.netcraft.com/site_report"><b>Netcraft</b></a> - detailed report about the site, helping you to make informed choices about their integrity.<b>*</b><br>
&nbsp;&nbsp; <a href="https://atlas.ripe.net/"><b>RIPE NCC Atlas</b></a> - a global, open, distributed Internet measurement platform.<br>
&nbsp;&nbsp; <a href="https://www.robtex.com/"><b>Robtex</b></a> - uses various sources to gather public information about IP numbers, domain names, host names, etc.<br>
&nbsp;&nbsp; <a href="https://securitytrails.com/"><b>Security Trails</b></a> - APIs for Security Companies, Researchers and Teams.<br>
&nbsp;&nbsp; <a href="https://tools.keycdn.com/curl"><b>Online Curl</b></a> - curl test, analyze HTTP Response Headers.<br>
&nbsp;&nbsp; <a href="https://extendsclass.com/"><b>Online Tools for Developers</b></a> - HTTP API tools, testers, encoders, converters, formatters, and other tools.<br>
&nbsp;&nbsp; <a href="https://ping.eu/"><b>Ping.eu</b></a> - online Ping, Traceroute, DNS lookup, WHOIS and others.<br>
&nbsp;&nbsp; <a href="https://network-tools.com/"><b>Network-Tools</b></a> - network tools for webmasters, IT technicians & geeks.<br>
&nbsp;&nbsp; <a href="https://bgpview.io/"><b>BGPview</b></a> - search for any ASN, IP, Prefix or Resource name.<br>
&nbsp;&nbsp; <a href="https://isbgpsafeyet.com/"><b>Is BGP safe yet?</b></a> - check BGP (RPKI) security of ISPs and other major Internet players.<br>
&nbsp;&nbsp; <a href="https://riseup.net/"><b>Riseup</b></a> - provides online communication tools for people and groups working on liberatory social change.<br>
&nbsp;&nbsp; <a href="https://www.virustotal.com/gui/home/upload"><b>VirusTotal</b></a> - analyze suspicious files and URLs to detect types of malware.<br>
</p>
##### :black_small_square: Privacy
<p>
&nbsp;&nbsp; <a href="https://www.privacyguides.org/"><b>privacyguides.org</b></a> - provides knowledge and tools to protect your privacy against global mass surveillance.<br>
&nbsp;&nbsp; <a href="https://dnsprivacy.org/wiki/display/DP/DNS+Privacy+Test+Servers"><b>DNS Privacy Test Servers</b></a> - DNS privacy recursive servers list (with a 'no logging' policy).<br>
</p>
##### :black_small_square: Code parsers/playgrounds
<p>
&nbsp;&nbsp; <a href="https://www.shellcheck.net/"><b>ShellCheck</b></a> - finds bugs in your shell scripts.<br>
&nbsp;&nbsp; <a href="https://explainshell.com/"><b>explainshell</b></a> - get interactive help texts for shell commands.<br>
&nbsp;&nbsp; <a href="https://jsbin.com/?html,output"><b>jsbin</b></a> - live pastebin for HTML, CSS & JavaScript, and more.<br>
&nbsp;&nbsp; <a href="https://codesandbox.io/"><b>CodeSandbox</b></a> - online code editor for web application development.<br>
&nbsp;&nbsp; <a href="http://sandbox.onlinephpfunctions.com/"><b>PHP Sandbox</b></a> - test your PHP code with this code tester.<br>
&nbsp;&nbsp; <a href="https://www.repl.it/"><b>Repl.it</b></a> - an instant IDE to learn, build, collaborate, and host all in one place.<br>
&nbsp;&nbsp; <a href="http://www.vclfiddle.net/"><b>vclFiddle</b></a> - is an online tool for experimenting with the Varnish Cache VCL.<br>
&nbsp;&nbsp; <a href="https://github.com/hadolint/hadolint"><b>Haskell Dockerfile Linter</b></a> - a smarter Dockerfile linter that helps you build best practice Docker images.<br>
</p>
##### :black_small_square: Performance
<p>
&nbsp;&nbsp; <a href="https://gtmetrix.com/"><b>GTmetrix</b></a> - analyze your sites speed and make it faster.<br>
&nbsp;&nbsp; <a href="https://performance.sucuri.net/"><b>Sucuri loadtimetester</b></a> - test here the
performance of any of your sites from across the globe.<br>
&nbsp;&nbsp; <a href="https://tools.pingdom.com/"><b>Pingdom Tools</b></a> - analyze your sites speed around the world.<br>
&nbsp;&nbsp; <a href="https://pingme.io/"><b>PingMe.io</b></a> - run website latency tests across multiple geographic regions.<br>
&nbsp;&nbsp; <a href="https://developers.google.com/speed/pagespeed/insights/"><b>PageSpeed Insights</b></a> - analyze your sites speed and make it faster.<br>
&nbsp;&nbsp; <a href="https://web.dev/"><b>web.dev</b></a> - helps developers like you learn and apply the web's modern capabilities to your own sites and apps.<br>
&nbsp;&nbsp; <a href="https://github.com/GoogleChrome/lighthouse"><b>Lighthouse</b></a> - automated auditing, performance metrics, and best practices for the web.<br>
</p>
##### :black_small_square: Mass scanners (search engines)
<p>
&nbsp;&nbsp; <a href="https://censys.io/"><b>Censys</b></a> - platform that helps information security practitioners discover, monitor, and analyze devices.<br>
&nbsp;&nbsp; <a href="https://www.shodan.io/"><b>Shodan</b></a> - the world's first search engine for Internet-connected devices.<br>
&nbsp;&nbsp; <a href="https://2000.shodan.io/#/"><b>Shodan 2000</b></a> - this tool looks for randomly generated data from Shodan.<br>
&nbsp;&nbsp; <a href="https://viz.greynoise.io/table"><b>GreyNoise</b></a> - mass scanner such as Shodan and Censys.<br>
&nbsp;&nbsp; <a href="https://www.zoomeye.org/"><b>ZoomEye</b></a> - search engine for cyberspace that lets the user find specific network components.<br>
&nbsp;&nbsp; <a href="https://netograph.io/"><b>netograph</b></a> - tools to monitor and understand deep structure of the web.<br>
&nbsp;&nbsp; <a href="https://fofa.so/"><b>FOFA</b></a> - is a cyberspace search engine.<br>
&nbsp;&nbsp; <a href="https://www.onyphe.io/"><b>onyphe</b></a> - is a search engine for open-source and cyber threat intelligence data collected.<br>
&nbsp;&nbsp; <a href="https://intelx.io/"><b>IntelligenceX</b></a> - is a search engine and data archive.<br>
&nbsp;&nbsp; <a href="https://app.binaryedge.io/"><b>binaryedge</b></a> - it scan the entire internet space and create real-time threat intelligence streams and reports.<br>
&nbsp;&nbsp; <a href="https://spyse.com/"><b>Spyse</b></a> - Internet assets registry: networks, threats, web objects, etc.<br>
&nbsp;&nbsp; <a href="https://wigle.net/"><b>wigle</b></a> - is a submission-based catalog of wireless networks. All the networks. Found by Everyone.<br>
&nbsp;&nbsp; <a href="https://publicwww.com/"><b>PublicWWW</b></a> - find any alphanumeric snippet, signature or keyword in the web pages HTML, JS and CSS code.<br>
&nbsp;&nbsp; <a href="https://inteltechniques.com/index.html"><b>IntelTechniques</b></a> - this repository contains hundreds of online search utilities.<br>
&nbsp;&nbsp; <a href="https://hunter.io/"><b>hunter</b></a> - lets you find email addresses in seconds and connect with the people that matter for your business.<br>
&nbsp;&nbsp; <a href="https://ghostproject.fr/"><b>GhostProject?</b></a> - search by full email address or username.<br>
&nbsp;&nbsp; <a href="https://www.databreaches.live/"><b>databreaches</b></a> - was my email affected by data breach?<br>
&nbsp;&nbsp; <a href="https://weleakinfo.com"><b>We Leak Info</b></a> - world's fastest and largest data breach search engine.<br>
&nbsp;&nbsp; <a href="https://pulsedive.com/"><b>Pulsedive</b></a> - scans of malicious URLs, IPs, and domains, including port scans and web requests.<br>
&nbsp;&nbsp; <a href="https://buckets.grayhatwarfare.com/"><b>Buckets by Grayhatwarfar</b></a> - database with public search for Open Amazon S3 Buckets and their contents.<br>
&nbsp;&nbsp; <a href="https://vigilante.pw/"><b>Vigilante.pw</b></a> - the breached database directory.<br>
&nbsp;&nbsp; <a href="https://builtwith.com/"><b>builtwith</b></a> - find out what websites are built with.<br>
&nbsp;&nbsp; <a href="https://nerdydata.com/"><b>NerdyData</b></a> - search the web's source code for technologies, across millions of sites.<br>
&nbsp;&nbsp; <a href="http://zorexeye.com/"><b>zorexeye</b></a> - search for sites, images, apps, softwares & more.<br>
&nbsp;&nbsp; <a href="https://www.mmnt.net/"><b>Mamont's open FTP Index</b></a> - if a target has an open FTP site with accessible content it will be listed here.<br>
&nbsp;&nbsp; <a href="https://osintframework.com/"><b>OSINT Framework</b></a> - focused on gathering information from free tools or resources.<br>
&nbsp;&nbsp; <a href="https://www.maltiverse.com/search"><b>maltiverse</b></a> - is a service oriented to cybersecurity analysts.<br>
&nbsp;&nbsp; <a href="https://leakedsource.ru/main/"><b>Leaked Source</b></a> - is a collaboration of data found online in the form of a lookup.<br>
&nbsp;&nbsp; <a href="https://search.weleakinfo.com/"><b>We Leak Info</b></a> - to help everyday individuals secure their online life, avoiding getting hacked.<br>
&nbsp;&nbsp; <a href="https://pipl.com/"><b>pipl</b></a> - is the place to find the person behind the email address, social username or phone number.<br>
&nbsp;&nbsp; <a href="https://abuse.ch/"><b>abuse.ch</b></a> - is operated by a random swiss guy fighting malware for non-profit.<br>
&nbsp;&nbsp; <a href="http://malc0de.com/database/"><b>malc0de</b></a> - malware search engine.<br>
&nbsp;&nbsp; <a href="https://cybercrime-tracker.net/index.php"><b>Cybercrime Tracker</b></a> - monitors and tracks various malware families that are used to perpetrate cyber crimes.<br>
&nbsp;&nbsp; <a href="https://github.com/eth0izzle/shhgit/"><b>shhgit</b></a> - find GitHub secrets in real time.<br>
&nbsp;&nbsp; <a href="https://searchcode.com/"><b>searchcode</b></a> - helping you find real world examples of functions, API's and libraries.<br>
&nbsp;&nbsp; <a href="http://www.insecam.org/"><b>Insecam</b></a> - the world biggest directory of online surveillance security cameras.<br>
&nbsp;&nbsp; <a href="http://index-of.es/"><b>index-of</b></a> - contains great stuff like: security, hacking, reverse engineering, cryptography, programming etc.<br>
&nbsp;&nbsp; <a href="https://opendata.rapid7.com/"><b>Rapid7 Labs Open Data</b></a> - is a great resources of datasets from Project Sonar.<br>
&nbsp;&nbsp; <a href="https://webtechsurvey.com/common-response-headers"><b>Common Response Headers</b></a> - the largest database of HTTP response headers.<br>
&nbsp;&nbsp; <a href="https://labs.inquest.net"><b>InQuest Labs</b></a> - InQuest Labs is an open, interactive, and API driven data portal for security researchers.<br>
</p>
##### :black_small_square: Generators
<p>
&nbsp;&nbsp; <a href="https://thispersondoesnotexist.com/"><b>thispersondoesnotexist</b></a> - generate fake faces in one click - endless possibilities.<br>
&nbsp;&nbsp; <a href="https://generated.photos"><b>AI Generated Photos</b></a> - 100.000 AI generated faces.<br>
&nbsp;&nbsp; <a href="https://www.fakenamegenerator.com/"><b>fakenamegenerator</b></a> - your randomly generated identity.<br>
&nbsp;&nbsp; <a href="https://tools.intigriti.io/redirector/"><b>Intigriti Redirector</b></a> - open redirect/SSRF payload generator.<br>
</p>
##### :black_small_square: Passwords
<p>
&nbsp;&nbsp; <a href="https://haveibeenpwned.com/"><b>have i been pwned?</b></a> - check if you have an account that has been compromised in a data breach.<br>
&nbsp;&nbsp; <a href="https://www.dehashed.com/"><b>dehashed</b></a> - is a hacked database search engine.<br>
&nbsp;&nbsp; <a href="https://leakedsource.ru/"><b>Leaked Source</b></a> - is a collaboration of data found online in the form of a lookup.<br>
</p>
##### :black_small_square: CVE/Exploits databases
<p>
&nbsp;&nbsp; <a href="https://cve.mitre.org/"><b>CVE Mitre</b></a> - list of publicly known cybersecurity vulnerabilities.<br>
&nbsp;&nbsp; <a href="https://www.cvedetails.com/"><b>CVE Details</b></a> - CVE security vulnerability advanced database.<br>
&nbsp;&nbsp; <a href="https://www.exploit-db.com/"><b>Exploit DB</b></a> - CVE compliant archive of public exploits and corresponding vulnerable software.<br>
&nbsp;&nbsp; <a href="https://0day.today/"><b>0day.today</b></a> - exploits market provides you the possibility to buy/sell zero-day exploits.<br>
&nbsp;&nbsp; <a href="https://sploitus.com/"><b>sploitus</b></a> - the exploit and tools database.<br>
&nbsp;&nbsp; <a href="https://cxsecurity.com/exploit/"><b>cxsecurity</b></a> - free vulnerability database.<br>
&nbsp;&nbsp; <a href="https://www.vulncode-db.com/"><b>Vulncode-DB</b></a> - is a database for vulnerabilities and their corresponding source code if available.<br>
&nbsp;&nbsp; <a href="https://cveapi.com/"><b>cveapi</b></a> - free API for CVE data.<br>
</p>
##### :black_small_square: Mobile apps scanners
<p>
&nbsp;&nbsp; <a href="https://www.immuniweb.com/mobile/"><b>ImmuniWeb® Mobile App Scanner</b></a> - test security and privacy of mobile apps (iOS & Android).<br>
&nbsp;&nbsp; <a href="https://vulnerabilitytest.quixxi.com/"><b>Quixxi</b></a> - free Mobile App Vulnerability Scanner for Android & iOS.<br>
&nbsp;&nbsp; <a href="https://www.ostorlab.co/scan/mobile/"><b>Ostorlab</b></a> - analyzes mobile application to identify vulnerabilities and potential weaknesses.<br>
</p>
##### :black_small_square: Private Search Engines
<p>
&nbsp;&nbsp; <a href="https://www.startpage.com/"><b>Startpage</b></a> - the world's most private search engine.<br>
&nbsp;&nbsp; <a href="https://searx.me/"><b>searX</b></a> - a privacy-respecting, hackable metasearch engine.<br>
&nbsp;&nbsp; <a href="https://darksearch.io/"><b>darksearch</b></a> - the 1st real Dark Web search engine.<br>
&nbsp;&nbsp; <a href="https://www.qwant.com/"><b>Qwant</b></a> - the search engine that respects your privacy.<br>
&nbsp;&nbsp; <a href="https://duckduckgo.com/"><b>DuckDuckGo</b></a> - the search engine that doesn't track you.<br>
&nbsp;&nbsp; <a href="https://swisscows.com/"><b>Swisscows</b></a> - privacy safe web search<br>
&nbsp;&nbsp; <a href="https://search.disconnect.me/"><b>Disconnect</b></a> - the search engine that anonymizes your searches.<br>
&nbsp;&nbsp; <a href="https://metager.org/"><b>MetaGer</b></a> - the search engine that uses anonymous proxy and hidden Tor branches.<br>
</p>
##### :black_small_square: Secure Webmail Providers
<p>
&nbsp;&nbsp; <a href="https://countermail.com/"><b>CounterMail</b></a> - online email service, designed to provide maximum security and privacy.<br>
&nbsp;&nbsp; <a href="http://mail2tor.com/"><b>Mail2Tor</b></a> - is a Tor Hidden Service that allows anyone to send and receive emails anonymously.<br>
&nbsp;&nbsp; <a href="https://tutanota.com/"><b>Tutanota</b></a> - is the world's most secure email service and amazingly easy to use.<br>
&nbsp;&nbsp; <a href="https://protonmail.com/"><b>Protonmail</b></a> - is the world's largest secure email service, developed by CERN and MIT scientists.<br>
&nbsp;&nbsp; <a href="https://www.startmail.com/en/"><b>Startmail</b></a> - private & encrypted email made easy.<br>
</p>
##### :black_small_square: Crypto
<p>
&nbsp;&nbsp; <a href="https://keybase.io/"><b>Keybase</b></a> - it's open source and powered by public-key cryptography.<br>
</p>
##### :black_small_square: PGP Keyservers
<p>
&nbsp;&nbsp; <a href="https://keyserver.ubuntu.com/"><b>SKS OpenPGP Key server</b></a> - services for the SKS keyservers used by OpenPGP.<br>
</p>
#### Systems/Services &nbsp;[<sup>[TOC]</sup>](#anger-table-of-contents)
@@ -0,0 +1,31 @@
#!/bin/bash
# Refresh sysadmin-toolbox from The Book of Secret Knowledge repo
# Run periodically to keep references current
set -e
REPO_URL="https://github.com/trimstray/the-book-of-secret-knowledge.git"
TEMP_DIR="/tmp/tbsk-refresh-$$"
SKILL_DIR="${1:-$HOME/clawd-duke-leto/skills/sysadmin-toolbox}"
echo "🔄 Refreshing sysadmin-toolbox from upstream..."
# Clone fresh copy
git clone --depth 1 "$REPO_URL" "$TEMP_DIR" 2>/dev/null
cd "$TEMP_DIR"
# Extract sections
echo "📦 Extracting references..."
awk '/^#### Shell One-liners/,/^#### Shell Tricks/' README.md > "$SKILL_DIR/references/shell-oneliners.md"
awk '/^#### Shell Tricks/,/^#### Shell Functions/' README.md > "$SKILL_DIR/references/shell-tricks.md"
awk '/^#### CLI Tools/,/^#### GUI Tools/' README.md > "$SKILL_DIR/references/cli-tools.md"
awk '/^#### Web Tools/,/^#### Systems\/Services/' README.md > "$SKILL_DIR/references/web-tools.md"
awk '/^#### Hacking\/Penetration Testing/,/^#### Your daily knowledge/' README.md > "$SKILL_DIR/references/security-tools.md"
# Cleanup
rm -rf "$TEMP_DIR"
echo "✅ sysadmin-toolbox refreshed from upstream"
echo " Shell one-liners: $(wc -l < "$SKILL_DIR/references/shell-oneliners.md") lines"
+23
View File
@@ -0,0 +1,23 @@
{
"version": 1,
"skills": {
"devops-engineer": {
"source": "jeffallan/claude-skills",
"sourceType": "github",
"skillPath": "skills/devops-engineer/SKILL.md",
"computedHash": "b5fefdd43470cf510937c882440fb0759327adfe4a21b511829e545e5faa988e"
},
"reverse-proxy": {
"source": "bagelhole/devops-security-agent-skills",
"sourceType": "github",
"skillPath": "infrastructure/networking/reverse-proxy/SKILL.md",
"computedHash": "07f33fdda533634da44489c6320d67391f1a8e243dd3e06891700e4ea8af72c0"
},
"sysadmin-toolbox": {
"source": "jdrhyne/agent-skills",
"sourceType": "github",
"skillPath": "skills/sysadmin-toolbox/SKILL.md",
"computedHash": "5e8851b6deb583491dee43ea36389ddbb4cbefa3a0c39d89382a3e79a6f80de4"
}
}
}