Module 1.3: Istio Security & Troubleshooting

Цей контент ще не доступний вашою мовою.

Complexity: `[COMPLEX]`

Time to Complete: 50-60 minutes

Prerequisites

Before starting this module, you should have completed:

Module 1: Installation & Architecture — istiod, Envoy, sidecar injection
Module 2: Traffic Management — VirtualService, DestinationRule, Gateway
Basic understanding of TLS, JWT tokens, and RBAC concepts

What You’ll Be Able to Do

After completing this module, you will be able to:

Configure PeerAuthentication policies to enforce mTLS modes (STRICT, PERMISSIVE) across namespaces and workloads
Implement RequestAuthentication with JWT validation and AuthorizationPolicy rules that control service-to-service access
Diagnose mTLS handshake failures, rejected requests, and policy conflicts using istioctl authn tls-check and proxy logs
Apply a systematic troubleshooting workflow using istioctl analyze, proxy-config, and proxy-status to resolve mesh issues

Why This Module Matters

Security accounts for 15% of the ICA exam and Troubleshooting accounts for 10%. Together, that’s a quarter of your score. Security questions will ask you to configure mTLS policies, set up JWT authentication, and write authorization rules. Troubleshooting questions will give you a broken configuration and ask you to find and fix it.

In production, these are the skills that prevent breaches and reduce MTTR. A misconfigured PeerAuthentication can silently break service communication. A missing AuthorizationPolicy can expose internal APIs. And when things go wrong, istioctl analyze and istioctl proxy-config are often the only tools that can tell you why.

The Building Security Analogy

Istio security works like a modern office building. PeerAuthentication (mTLS) is the locked front door — it verifies everyone entering is who they claim to be (mutual certificate exchange). RequestAuthentication (JWT) is the badge reader — it validates the badge is legitimate but doesn’t decide who can enter which rooms. AuthorizationPolicy is the access control list — it decides which badge holders can open which doors. You need all three for complete security.

What You’ll Learn

By the end of this module, you’ll be able to:

Configure mTLS using PeerAuthentication (STRICT, PERMISSIVE, DISABLE)
Set up JWT validation with RequestAuthentication
Write fine-grained AuthorizationPolicy rules (ALLOW, DENY, CUSTOM)
Configure TLS at the ingress gateway
Use istioctl analyze, proxy-status, and proxy-config for debugging
Read Envoy logs and diagnose common issues
Fix broken Istio configurations systematically

Did You Know?

Istio rotates mTLS certificates every 24 hours by default: Each workload gets a short-lived SPIFFE certificate (spiffe://<trust-domain>/ns/<namespace>/sa/<service-account>) that’s automatically rotated. No manual cert management needed.
DENY policies are evaluated before ALLOW: Istio’s authorization engine processes DENY rules first, then ALLOW rules, then CUSTOM rules. A DENY match short-circuits — the request is rejected regardless of ALLOW rules.
istioctl analyze catches 40+ misconfiguration types: Including orphaned VirtualServices, missing DestinationRules, conflicting policies, and deprecated API versions. It’s the single most useful command for the ICA exam.

War Story: The Midnight mTLS Migration

Characters:

Marcus: Platform Engineer (4 years experience)
Team: 12 microservices on Kubernetes

The Incident:

Marcus was tasked with enabling mTLS across the entire mesh. He’d read the docs and knew STRICT mode was the goal. On a Thursday evening, he applied a mesh-wide PeerAuthentication:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT

Within 30 seconds, the monitoring dashboard lit up red. The payments service was calling a legacy service running outside the mesh (no sidecar). STRICT mTLS required both sides to present certificates. The legacy service didn’t have a sidecar, couldn’t present a certificate, and every request failed with connection reset by peer.

Orders stopped processing. The on-call engineer reverted the change 8 minutes later, but 2,400 orders were lost during the outage.

What Marcus should have done:

# Step 1: Start with PERMISSIVE (accepts both mTLS and plaintext)
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: PERMISSIVE

# Step 2: Identify services without sidecars
# istioctl proxy-status  (shows which pods have proxies)

# Step 3: Exclude specific ports or services
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT
  portLevelMtls:
    8080:
      mode: DISABLE    # Legacy service port

# Step 4: Or apply STRICT per-namespace, not mesh-wide
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: payments  # Only this namespace
spec:
  mtls:
    mode: STRICT

Lesson: Always start with PERMISSIVE, verify all services have sidecars with istioctl proxy-status, then progressively enable STRICT per-namespace.

Part 1: Mutual TLS (mTLS)

1.1 How mTLS Works in Istio

Without mTLS:
Pod A ──── plaintext HTTP ────► Pod B
         (anyone can intercept)

With mTLS:
Pod A                                    Pod B
┌──────────────┐                        ┌──────────────┐
│ App          │                        │ App          │
│  ↓           │                        │  ↑           │
│ Envoy Proxy  │◄── encrypted TLS ────►│ Envoy Proxy  │
│ (has cert A) │    (mutual verify)     │ (has cert B) │
└──────────────┘                        └──────────────┘

Both sides verify each other's identity via SPIFFE certificates
issued by istiod's built-in CA (Citadel).

Certificate identity: Each workload gets a SPIFFE identity:

spiffe://cluster.local/ns/default/sa/reviews
         └─ trust domain  └─ namespace  └─ service account

1.2 PeerAuthentication

PeerAuthentication controls mTLS behavior for workloads in the mesh.

Mesh-wide policy (namespace: istio-system):

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system        # Mesh-wide when in istio-system
spec:
  mtls:
    mode: STRICT                 # Require mTLS for all services

Namespace-level policy:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: payments            # Only affects this namespace
spec:
  mtls:
    mode: STRICT

Workload-level policy:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: reviews-mtls
  namespace: default
spec:
  selector:
    matchLabels:
      app: reviews               # Only affects pods with this label
  mtls:
    mode: STRICT

Port-level policy:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: reviews-mtls
  namespace: default
spec:
  selector:
    matchLabels:
      app: reviews
  mtls:
    mode: STRICT
  portLevelMtls:
    8080:
      mode: DISABLE              # Disable mTLS on port 8080 only

mTLS modes:

Mode	Behavior	Use Case
`STRICT`	Only accepts mTLS traffic	Production (full encryption)
`PERMISSIVE`	Accepts both mTLS and plaintext	Migration period
`DISABLE`	No mTLS	Legacy services, debugging
`UNSET`	Inherits from parent	Default behavior

Policy priority (most specific wins):

Workload-level  >  Namespace-level  >  Mesh-level
(selector)         (namespace)          (istio-system)

1.3 DestinationRule TLS Settings

PeerAuthentication controls the server side. DestinationRule controls the client side:

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: reviews
spec:
  host: reviews
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL          # Use Istio's mTLS certs

DestinationRule TLS modes:

Mode	Description
`DISABLE`	No TLS
`SIMPLE`	Originate TLS (client verifies server)
`MUTUAL`	Originate mTLS (both verify each other)
`ISTIO_MUTUAL`	Use Istio’s built-in mTLS certificates

Exam tip: In most cases, you don’t need to set DestinationRule TLS mode explicitly. Istio auto-detects when the destination has mTLS enabled and uses ISTIO_MUTUAL automatically. Only set it when you need to override behavior.

Part 2: Request Authentication (JWT)

RequestAuthentication validates JSON Web Tokens (JWT) attached to requests. It verifies the token is valid but does NOT enforce that a token is required — that’s AuthorizationPolicy’s job.

2.1 Basic JWT Validation

apiVersion: security.istio.io/v1
kind: RequestAuthentication
metadata:
  name: jwt-auth
  namespace: default
spec:
  selector:
    matchLabels:
      app: productpage
  jwtRules:
  - issuer: "https://accounts.google.com"
    jwksUri: "https://www.googleapis.com/oauth2/v3/certs"
  - issuer: "https://my-auth.example.com"
    jwksUri: "https://my-auth.example.com/.well-known/jwks.json"
    forwardOriginalToken: true     # Forward JWT to upstream
    outputPayloadToHeader: "x-jwt-payload"  # Extract claims to header

What RequestAuthentication does:

If a request has a JWT, validate it (issuer, signature, expiry)
If the JWT is invalid, reject the request (401)
If the request has NO JWT, allow it through (this surprises people!)

To require a valid JWT, you need an AuthorizationPolicy (see Part 3).

2.2 JWT with Claim-Based Routing

Extract JWT claims and use them for authorization or routing:

apiVersion: security.istio.io/v1
kind: RequestAuthentication
metadata:
  name: jwt-auth
  namespace: default
spec:
  selector:
    matchLabels:
      app: frontend
  jwtRules:
  - issuer: "https://auth.example.com"
    jwksUri: "https://auth.example.com/.well-known/jwks.json"
    outputClaimToHeaders:
    - header: x-jwt-sub
      claim: sub
    - header: x-jwt-groups
      claim: groups

Part 3: Authorization Policy

AuthorizationPolicy is Istio’s access control mechanism. It decides who can access what.

3.1 Policy Actions

Request arrives
     │
     ▼
┌─ CUSTOM policies ─┐  (if any, checked first via external authz)
│  Match? → delegate │
└────────────────────┘
     │
     ▼
┌─── DENY policies ──┐  (checked second)
│  Match? → REJECT   │
└─────────────────────┘
     │
     ▼
┌── ALLOW policies ──┐  (checked third)
│  Match? → ALLOW    │
│  No match? → DENY  │  ← If ANY allow policy exists, default is deny
└─────────────────────┘
     │
     ▼
  No policies? → ALLOW (default)

Critical: If there are NO AuthorizationPolicies for a workload, all traffic is allowed. The moment you create ANY ALLOW policy, all traffic that doesn’t match an ALLOW rule is denied.

3.2 ALLOW Policy

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: allow-reviews
  namespace: default
spec:
  selector:
    matchLabels:
      app: reviews
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/default/sa/productpage"]
    to:
    - operation:
        methods: ["GET"]
        paths: ["/reviews/*"]

This allows: GET requests to /reviews/* from the productpage service account. Everything else to the reviews service is denied.

3.3 DENY Policy

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: deny-external
  namespace: default
spec:
  selector:
    matchLabels:
      app: internal-api
  action: DENY
  rules:
  - from:
    - source:
        notNamespaces: ["default", "backend"]
    to:
    - operation:
        paths: ["/admin/*"]

This denies: Any request to /admin/* on the internal-api service from namespaces other than default or backend.

3.4 Require JWT (Combining RequestAuthentication + AuthorizationPolicy)

# Step 1: Validate JWT if present
apiVersion: security.istio.io/v1
kind: RequestAuthentication
metadata:
  name: require-jwt
  namespace: default
spec:
  selector:
    matchLabels:
      app: productpage
  jwtRules:
  - issuer: "https://auth.example.com"
    jwksUri: "https://auth.example.com/.well-known/jwks.json"
---
# Step 2: DENY requests without valid JWT
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: require-jwt
  namespace: default
spec:
  selector:
    matchLabels:
      app: productpage
  action: DENY
  rules:
  - from:
    - source:
        notRequestPrincipals: ["*"]   # No valid JWT principal = deny

3.5 Namespace-Level Policies

# Allow all traffic within the namespace
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: allow-same-namespace
  namespace: backend
spec:
  action: ALLOW
  rules:
  - from:
    - source:
        namespaces: ["backend"]
---
# Deny all traffic (explicit deny-all)
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: deny-all
  namespace: backend
spec:
  {}                               # Empty spec = deny all

3.6 Common AuthorizationPolicy Patterns

Allow specific HTTP methods:

rules:
- to:
  - operation:
      methods: ["GET", "HEAD"]

Allow from specific service accounts:

rules:
- from:
  - source:
      principals: ["cluster.local/ns/frontend/sa/webapp"]

Allow based on JWT claims:

rules:
- from:
  - source:
      requestPrincipals: ["https://auth.example.com/*"]
  when:
  - key: request.auth.claims[role]
    values: ["admin"]

Allow specific IP ranges:

rules:
- from:
  - source:
      ipBlocks: ["10.0.0.0/8"]

Part 4: TLS at Ingress

4.1 Simple TLS (Server Certificate Only)

# Create TLS secret
kubectl create -n istio-system secret tls my-tls-secret \
  --key=server.key \
  --cert=server.crt

apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: secure-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
    - "app.example.com"
    tls:
      mode: SIMPLE
      credentialName: my-tls-secret

4.2 Mutual TLS at Ingress (Client Certificates)

# Create secret with CA cert for client verification
kubectl create -n istio-system secret generic my-mtls-secret \
  --from-file=tls.key=server.key \
  --from-file=tls.crt=server.crt \
  --from-file=ca.crt=ca.crt

apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: mtls-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
    - "secure.example.com"
    tls:
      mode: MUTUAL                    # Require client certificate
      credentialName: my-mtls-secret

Part 5: Troubleshooting

5.1 istioctl analyze

The first command you should run when something isn’t working:

# Analyze all namespaces
istioctl analyze --all-namespaces

# Analyze specific namespace
istioctl analyze -n default

# Analyze a specific file before applying
istioctl analyze my-virtualservice.yaml

# Common warnings/errors:
# IST0101: Referenced host not found
# IST0104: Gateway references missing secret
# IST0106: Schema validation error
# IST0108: Unknown annotation
# IST0113: VirtualService references undefined subset

5.2 istioctl proxy-status

Check if proxies are connected and in sync with istiod:

istioctl proxy-status

Output interpretation:

NAME                              CDS    LDS    EDS    RDS    ECDS   ISTIOD
productpage-v1-xxx.default        SYNCED SYNCED SYNCED SYNCED SYNCED istiod-xxx
reviews-v1-xxx.default            SYNCED SYNCED SYNCED SYNCED SYNCED istiod-xxx
ratings-v1-xxx.default            STALE  SYNCED SYNCED SYNCED SYNCED istiod-xxx  ← Problem!

Status	Meaning	Action
`SYNCED`	Proxy has latest config from istiod	Normal
`NOT SENT`	istiod hasn’t sent config (no changes)	Usually normal
`STALE`	Proxy hasn’t acknowledged latest config	Investigate — restart pod or check connectivity

xDS types:

Type	Full Name	What It Configures
CDS	Cluster Discovery Service	Upstream clusters (services)
LDS	Listener Discovery Service	Inbound/outbound listeners
EDS	Endpoint Discovery Service	Endpoints (pod IPs)
RDS	Route Discovery Service	HTTP routing rules
ECDS	Extension Config Discovery	WASM extensions

5.3 istioctl proxy-config

Deep inspection of what Envoy is actually configured to do:

# List all clusters (upstream services) for a pod
istioctl proxy-config clusters productpage-v1-xxx.default

# List listeners (what ports Envoy is listening on)
istioctl proxy-config listeners productpage-v1-xxx.default

# List routes (HTTP routing rules)
istioctl proxy-config routes productpage-v1-xxx.default

# List endpoints (actual pod IPs)
istioctl proxy-config endpoints productpage-v1-xxx.default

# Show the full Envoy config dump
istioctl proxy-config all productpage-v1-xxx.default -o json

# Filter by specific service
istioctl proxy-config endpoints productpage-v1-xxx.default \
  --cluster "outbound|9080||reviews.default.svc.cluster.local"

5.4 Envoy Access Logs

Enable access logs to see every request flowing through the mesh:

# Enable via mesh config
istioctl install --set meshConfig.accessLogFile=/dev/stdout -y

# View logs for a specific pod's sidecar
kubectl logs productpage-v1-xxx -c istio-proxy

# Sample log entry:
# [2024-01-15T10:30:00.000Z] "GET /reviews/1 HTTP/1.1" 200 - via_upstream
#   - 0 325 45 42 "-" "curl/7.68.0" "xxx" "reviews:9080"
#   "10.244.0.15:9080" outbound|9080||reviews.default.svc.cluster.local
#   10.244.0.10:50542 10.96.10.15:9080 10.244.0.10:50540

Log format breakdown:

[timestamp] "METHOD PATH PROTOCOL" STATUS_CODE FLAGS
  - REQUEST_BYTES RESPONSE_BYTES DURATION_MS UPSTREAM_DURATION
  "USER_AGENT" "REQUEST_ID" "AUTHORITY"
  "UPSTREAM_HOST" UPSTREAM_CLUSTER
  DOWNSTREAM_LOCAL DOWNSTREAM_REMOTE DOWNSTREAM_PEER

5.5 Common Issues and Fixes

Issue	Symptoms	Diagnostic	Fix
Missing sidecar	Service not in mesh, no mTLS	`kubectl get pod -o jsonpath='{.spec.containers[*].name}'`	Label namespace + restart pods
VirtualService not applied	Traffic ignores routing rules	`istioctl analyze` (IST0113)	Check hosts match, gateway reference exists
mTLS STRICT with non-mesh service	`connection reset by peer`	`istioctl proxy-status` (missing pod)	Use PERMISSIVE or add sidecar
Stale proxy config	Old routing rules in effect	`istioctl proxy-status` (STALE)	Restart the pod
Gateway TLS misconfigured	TLS handshake failure	`istioctl analyze` (IST0104)	Check credentialName matches K8s Secret
AuthorizationPolicy blocking	403 Forbidden	`kubectl logs <pod> -c istio-proxy`	Check RBAC filters in access logs
Subset not defined	503 `no healthy upstream`	`istioctl analyze` (IST0113)	Create DestinationRule with matching subsets
Port name wrong	Protocol detection fails	`kubectl get svc -o yaml` (check port names)	Name ports as `http-xxx`, `grpc-xxx`, `tcp-xxx`

5.6 Debugging Workflow

When something isn’t working in the mesh, follow this systematic approach:

Step 1: istioctl analyze -n <namespace>
        → Catches 80% of misconfigurations

Step 2: istioctl proxy-status
        → Is the proxy connected? Is config synced?

Step 3: istioctl proxy-config routes <pod>
        → Does the proxy have the expected routing rules?

Step 4: kubectl logs <pod> -c istio-proxy
        → What does the access log show? 4xx? 5xx? Timeout?

Step 5: istioctl proxy-config clusters <pod>
        → Can the proxy see the upstream service?

Step 6: istioctl proxy-config endpoints <pod> --cluster <cluster>
        → Are there healthy endpoints?

                  Debugging Decision Tree

                 Something isn't working
                         │
                         ▼
              Run istioctl analyze
                   │          │
              Issues found    No issues
                   │          │
              Fix them        ▼
                         Is sidecar injected?
                           │          │
                          No          Yes
                           │          │
                    Inject it         ▼
                                proxy-status SYNCED?
                                  │          │
                                 No          Yes
                                  │          │
                           Restart pod       ▼
                                       Check access logs
                                         │          │
                                       4xx/5xx    No logs
                                         │          │
                                    Policy/route  Traffic not
                                    issue        reaching proxy

Common Mistakes

Mistake	Symptom	Solution
STRICT mTLS with non-mesh services	Connection refused/reset	Use PERMISSIVE mode or add sidecars
RequestAuthentication without AuthorizationPolicy	Unauthenticated requests pass through	Add DENY policy for `notRequestPrincipals: ["*"]`
Creating ALLOW policy without catch-all	All non-matching traffic denied (surprise!)	Understand that any ALLOW policy = default deny
Empty AuthorizationPolicy spec	All traffic denied	`spec: {}` means deny-all, add rules for allowed traffic
Wrong namespace for mesh-wide policy	Policy only applies to one namespace	Mesh-wide PeerAuthentication must be in `istio-system`
Forgetting `credentialName` on Gateway TLS	TLS handshake fails	Create K8s Secret in `istio-system` namespace
Not checking port naming conventions	Protocol detection fails, policies don’t apply	Name Service ports: `http-`, `grpc-`, `tcp-*`
Ignoring DENY-before-ALLOW evaluation order	ALLOW policy seems to not work	Check if a DENY policy is matching first

Quiz

Q1: What is the difference between STRICT and PERMISSIVE mTLS?

Show Answer

STRICT: Only accepts mTLS-encrypted traffic. Plaintext connections are rejected. Use when all communicating services have sidecars.
PERMISSIVE: Accepts both mTLS and plaintext traffic. Use during migration when some services don’t have sidecars yet.

PERMISSIVE is the default mode. Always start with PERMISSIVE and graduate to STRICT after verifying all services have sidecars.

Q2: You create a RequestAuthentication for a service. A request arrives without any JWT token. What happens?

Show Answer

The request is allowed through. RequestAuthentication only validates tokens that are present — it does NOT require them. To reject requests without a valid JWT, add an AuthorizationPolicy:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: require-jwt
spec:
  selector:
    matchLabels:
      app: myservice
  action: DENY
  rules:
  - from:
    - source:
        notRequestPrincipals: ["*"]

Q3: What is the evaluation order for AuthorizationPolicy actions?

Show Answer

CUSTOM (external authorization) — checked first
DENY — checked second, short-circuits on match
ALLOW — checked third
If no policies exist → all traffic allowed
If ALLOW policies exist but none match → traffic denied

Q4: Write an AuthorizationPolicy that allows only the frontend service account to call the backend service via GET on /api/*.

Show Answer

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: backend-policy
  namespace: default
spec:
  selector:
    matchLabels:
      app: backend
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/default/sa/frontend"]
    to:
    - operation:
        methods: ["GET"]
        paths: ["/api/*"]

Q5: A service returns connection reset by peer after enabling STRICT mTLS. What is the most likely cause?

Show Answer

The calling service (or the target) doesn’t have an Envoy sidecar. STRICT mode requires both sides to present mTLS certificates. Without a sidecar, the service can’t participate in mTLS.

Diagnostic steps:

istioctl proxy-status — check if both pods appear
kubectl get pod <pod> -o jsonpath='{.spec.containers[*].name}' — look for istio-proxy
Fix: inject the sidecar, or use PERMISSIVE mode for that workload

Q6: What does istioctl proxy-status show, and what does STALE mean?

Show Answer

istioctl proxy-status shows the synchronization state between istiod and every Envoy proxy in the mesh. For each proxy, it shows the status of 5 xDS types: CDS, LDS, EDS, RDS, ECDS.

SYNCED: Proxy has the latest configuration
NOT SENT: No config changes to send (normal)
STALE: istiod sent config but the proxy hasn’t acknowledged it — indicates a problem (network issue, overloaded proxy)

Fix STALE: restart the affected pod.

Q7: How do you enable Envoy access logging for all sidecars?

Show Answer

# During installation
istioctl install --set meshConfig.accessLogFile=/dev/stdout -y

# Or via IstioOperator
spec:
  meshConfig:
    accessLogFile: /dev/stdout

Then view logs with:

kubectl logs <pod-name> -c istio-proxy

Q8: What is the command to see all routing rules configured in a specific pod’s Envoy proxy?

Show Answer

istioctl proxy-config routes <pod-name>.<namespace>

This shows the Route Discovery Service (RDS) configuration — all HTTP routes the proxy knows about. To see more detail:

istioctl proxy-config routes <pod-name>.<namespace> -o json

For other config types: clusters, listeners, endpoints, all.

Q9: You applied an AuthorizationPolicy with action: ALLOW, but now ALL traffic to the service is blocked except what matches the rule. Why?

Show Answer

This is by design. When any ALLOW policy exists for a workload, the default behavior becomes deny-all for that workload. Only traffic that explicitly matches an ALLOW rule is permitted.

If you want to allow additional traffic patterns, either:

Add more rules to the existing ALLOW policy
Create additional ALLOW policies
Remove the ALLOW policy if you want default-allow behavior

Q10: How do you debug why a VirtualService routing rule is not being applied?

Show Answer

Systematic approach:

istioctl analyze -n <ns> — Check for IST0113 (missing subset), IST0101 (host not found)
istioctl proxy-status — Verify proxy is SYNCED
istioctl proxy-config routes <pod> — Check if the route appears in Envoy’s config
kubectl logs <pod> -c istio-proxy — Check access logs for actual routing behavior
Verify hosts match — VirtualService hosts must match the service name or Gateway host
Check gateways field — If using Gateway, VirtualService must reference it
Check namespace — VirtualService must be in the same namespace as the service (or use exportTo)

Hands-On Exercise: Security & Troubleshooting

Objective

Configure mTLS, authorization policies, and practice troubleshooting common Istio issues.

Setup

# Ensure Istio is installed with demo profile
istioctl install --set profile=demo \
  --set meshConfig.accessLogFile=/dev/stdout -y

kubectl label namespace default istio-injection=enabled --overwrite

# Deploy Bookinfo
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.22/samples/bookinfo/platform/kube/bookinfo.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.22/samples/bookinfo/networking/destination-rule-all.yaml

kubectl wait --for=condition=ready pod --all -n default --timeout=120s

Task 1: Enable STRICT mTLS

# Apply mesh-wide STRICT mTLS
kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT
EOF

# Verify mTLS is working
istioctl proxy-config clusters productpage-v1-$(kubectl get pods -l app=productpage -o jsonpath='{.items[0].metadata.name}' | cut -d'-' -f3-) | grep reviews

Verify: Traffic should still work between all services since they all have sidecars.

kubectl exec $(kubectl get pod -l app=ratings -o jsonpath='{.items[0].metadata.name}') -c ratings -- curl -s productpage:9080/productpage | head -20

Task 2: Create Authorization Policies

# Deny all traffic to reviews (start restrictive)
kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: deny-all-reviews
  namespace: default
spec:
  selector:
    matchLabels:
      app: reviews
  action: DENY
  rules:
  - from:
    - source:
        notPrincipals: ["cluster.local/ns/default/sa/bookinfo-productpage"]
EOF

Verify: Only productpage can reach reviews. Requests from other services should get 403:

# This should work (productpage → reviews)
kubectl exec $(kubectl get pod -l app=productpage -o jsonpath='{.items[0].metadata.name}') \
  -c productpage -- curl -s -o /dev/null -w "%{http_code}" http://reviews:9080/reviews/1

# This should fail with 403 (ratings → reviews)
kubectl exec $(kubectl get pod -l app=ratings -o jsonpath='{.items[0].metadata.name}') \
  -c ratings -- curl -s -o /dev/null -w "%{http_code}" http://reviews:9080/reviews/1

Task 3: Troubleshooting Practice

Intentionally break something and fix it:

# Create a VirtualService with a typo in the subset name
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: reviews-broken
spec:
  hosts:
  - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v99   # This subset doesn't exist!
EOF

# Now diagnose:
# Step 1: Analyze
istioctl analyze -n default
# Expected: IST0113 - Referenced subset not found

# Step 2: Check proxy config
istioctl proxy-config routes $(kubectl get pod -l app=productpage \
  -o jsonpath='{.items[0].metadata.name}').default | grep reviews

# Step 3: Fix it
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: reviews-broken
spec:
  hosts:
  - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v1    # Fixed!
EOF

# Step 4: Verify
istioctl analyze -n default

Task 4: Inspect Envoy Configuration

# Get the productpage pod name
PP_POD=$(kubectl get pod -l app=productpage -o jsonpath='{.items[0].metadata.name}')

# View all clusters (upstream services)
istioctl proxy-config clusters $PP_POD.default

# View listeners
istioctl proxy-config listeners $PP_POD.default

# View routes
istioctl proxy-config routes $PP_POD.default

# View endpoints for reviews service
istioctl proxy-config endpoints $PP_POD.default \
  --cluster "outbound|9080||reviews.default.svc.cluster.local"

# Check access logs
kubectl logs $PP_POD -c istio-proxy --tail=10

Success Criteria

STRICT mTLS is enabled mesh-wide and all services communicate successfully
AuthorizationPolicy correctly restricts reviews access to productpage only
You can identify the IST0113 error from istioctl analyze for the broken VirtualService
You can use proxy-config to inspect clusters, listeners, routes, and endpoints
Access logs show request details in the istio-proxy container

Cleanup

kubectl delete peerauthentication default -n istio-system
kubectl delete authorizationpolicy deny-all-reviews -n default
kubectl delete virtualservice reviews-broken -n default
kubectl delete -f https://raw.githubusercontent.com/istio/istio/release-1.22/samples/bookinfo/platform/kube/bookinfo.yaml
kubectl delete -f https://raw.githubusercontent.com/istio/istio/release-1.22/samples/bookinfo/networking/destination-rule-all.yaml
istioctl uninstall --purge -y
kubectl delete namespace istio-system

Next Module

Continue to Module 4: Istio Observability to learn about Istio metrics, distributed tracing, access logging, and dashboards with Kiali and Grafana. Observability is 10% of the ICA exam.

Final Exam Prep Checklist

Good luck on your ICA exam!

Module 1.3: Istio Security & Troubleshooting

Complexity: [COMPLEX]

Time to Complete: 50-60 minutes

Prerequisites

What You’ll Be Able to Do

Why This Module Matters

What You’ll Learn

Did You Know?

War Story: The Midnight mTLS Migration

Part 1: Mutual TLS (mTLS)

1.1 How mTLS Works in Istio

1.2 PeerAuthentication

1.3 DestinationRule TLS Settings

Part 2: Request Authentication (JWT)

2.1 Basic JWT Validation

2.2 JWT with Claim-Based Routing

Part 3: Authorization Policy

3.1 Policy Actions

3.2 ALLOW Policy

3.3 DENY Policy

3.4 Require JWT (Combining RequestAuthentication + AuthorizationPolicy)

3.5 Namespace-Level Policies

3.6 Common AuthorizationPolicy Patterns

Part 4: TLS at Ingress

4.1 Simple TLS (Server Certificate Only)

4.2 Mutual TLS at Ingress (Client Certificates)

Part 5: Troubleshooting

5.1 istioctl analyze

5.2 istioctl proxy-status

5.3 istioctl proxy-config

5.4 Envoy Access Logs

5.5 Common Issues and Fixes

5.6 Debugging Workflow

Common Mistakes

Quiz

Hands-On Exercise: Security & Troubleshooting

Objective

Setup

Task 1: Enable STRICT mTLS

Task 2: Create Authorization Policies

Task 3: Troubleshooting Practice

Task 4: Inspect Envoy Configuration

Success Criteria

Cleanup

Next Module

Final Exam Prep Checklist

Complexity: `[COMPLEX]`