Lock Down Permissions and Make Production Debuggable

Over-scoped bots are security incidents waiting to happen. When your bot has users:read.email but never uses it, you're one compromised token away from a data breach. Meanwhile, unstructured console.log statements are useless when debugging production issues during a critical customer demo. You need minimal scopes and searchable logs.

Outcome

Create a scope truth table, trim unused permissions, and implement structured logging that tracks every AI operation with searchable fields.

Fast Track

1. Generate scope truth table from manifest and code usage
2. Remove unused scopes and reinstall to verify failures
3. Replace all console.log with structured app.logger calls

Building on Previous Lessons

We're extending the observability foundation to production scale:

• From correlation middleware: Correlation middleware forms the base - now we add AI-specific metrics (model, tokens, latency)
• From repository flyover: Context fetching operations get structured logging to track API quotas and rate limits
• From AI tools: Tool executions log inputs and outputs for debugging AI decision chains
• From status communication: Status updates become searchable events for measuring user-perceived latency
• From error handling: Retry attempts, backoff delays, and model fallbacks are tracked for reliability metrics
• Production reasoning: Distributed serverless functions need structured logs - correlation IDs are the only way to trace a request across multiple function executions
• Cost tracking: Token counts in logs enable chargeback models (per-user/per-channel cost attribution)

Hands-On Exercise 5.2

Audit your bot's OAuth scopes and implement production-grade structured logging:

Requirements:

1. Create a scope truth table mapping each scope to its usage
2. Remove all unused scopes from manifest
3. Test with removed scopes to confirm expected failures
4. Implement structured logging schema for all AI operations
5. Include correlation ID, tokens, latency, and retry info

Implementation hints:

• Use ast-grep to find all Slack API calls and their required scopes
• Test scope removal by commenting out, reinstalling, and verifying failure
• Add structured fields to every app.logger call
• Include both prompt and completion tokens for cost tracking

Logging schema to implement:

interface StructuredLog {
  correlationId: string;
  operation: 'respondToMessage' | 'toolCall' | 'retry';
  model?: string;
  promptTokens?: number;
  completionTokens?: number;
  retryAttempt?: number;
  rateLimitWaitMs?: number;
  channel?: string;
  thread_ts?: string;
  latencyMs?: number;
  error?: string;
}

Try It

1. Generate scope truth table:

   # Find all Slack API calls
   ast-grep --pattern 'client.$METHOD($$$)' server/

   Create /slack-agent/SCOPE_AUDIT.md:

   # OAuth Scope Audit
    
   | Scope | Used By | Can Remove? |
   |-------|---------|-------------|
   | app_mentions:read | app-mention.ts:12 - Event subscription | ❌ Required |
   | channels:history | respond-to-message.ts:34 - Thread context | ❌ Required |
   | chat:write | All responses | ❌ Required |
   | groups:history | Private channel support | ✅ Not used |
   | im:history | DM support | ❌ Required |
   | mpim:history | Group DM support | ✅ Not used |
   | users:read | None found | ✅ Remove |
   | users:read.email | None found | ✅ Remove |

2. Test scope removal:

   /slack-agent/manifest.json

   "bot": [
     "app_mentions:read",
     "channels:history",
     "chat:write",
     "im:history"
     // Removed: groups:history, mpim:history, users:read
   ]

   After reinstall, test in private channel:

   [ERROR] bolt-app {
     correlationId: 'ev09E5_1234567890.123456',
     error: 'missing_scope',
     required: 'groups:history',
     channel: 'G09D4DG727P'
   } Cannot read private channel history
   

3. Implement structured logging:

   /slack-agent/server/lib/ai/respond-to-message.ts

   const result = await generateText({
     model: 'openai/gpt-5.4-mini',
     instructions: buildSystemPrompt(event),
     messages,
     stopWhen: isStepCount(5),
   });
    
   app.logger.info({
     ...context.correlation,
     operation: 'respondToMessage',
     model: 'openai/gpt-5.4-mini',
     promptTokens: result.usage.inputTokens,
     completionTokens: result.usage.outputTokens,
     retryAttempt: retryCount,
     rateLimitWaitMs: rateLimitDelay,
     channel: event.channel,
     thread_ts: event.thread_ts || event.ts,
     latencyMs: Date.now() - startTime
   }, 'AI response generated');

4. Verify structured logs in Vercel:

   {
     "level": 30,
     "time": 1733456789123,
     "msg": "AI response generated",
     "correlationId": "ev09E5EDA89M_1234567890.123456",
     "operation": "respondToMessage",
     "model": "openai/gpt-5.4-mini",
     "promptTokens": 234,
     "completionTokens": 89,
     "retryAttempt": 0,
     "rateLimitWaitMs": 0,
     "channel": "C09D4DG727P",
     "thread_ts": "1234567890.123456",
     "latencyMs": 1247
   }

5. Query logs by correlation ID:

   Paste the bare ID into the Logs search bar. Free text search matches the log message, and the structured logger writes the correlation ID into each message. Custom JSON keys like correlationId aren't filterable fields, so a correlationId:"..." filter won't work.

   Vercel Logs search: ev09E5EDA89M_1234567890.123456
   
   Results (4 entries):
   1. [INFO] Processing app_mention event
   2. [INFO] Thread context retrieved (7 messages)
   3. [INFO] Tool call: searchKnowledgeBase
   4. [INFO] AI response generated (1247ms)
   

Troubleshooting

Missing Scope Errors:

• Check exact error message for required scope
• Some scopes have dependencies (e.g., channels:history needs channels:read)
• Private channels need groups: scopes even if not explicitly used

Structured Logging Not Working:

• Ensure you're using app.logger not console.log
• Vercel may need NODE_ENV=production for proper log levels
• Check log level configuration in production

Token Counts Missing: For streaming responses, collect usage after stream completes:

const result = streamText({ /* same options as createTextStream */ });
for await (const text of result.textStream) {
  // append text to the Slack stream
}
const usage = await result.usage; // { inputTokens, outputTokens, totalTokens } across all steps

Commit

git add -A
git commit -m "feat(security): minimize OAuth scopes and add structured logging
 
- Create scope audit table with usage mapping
- Remove unused scopes: groups:history, mpim:history, users:read
- Implement structured logging schema with AI metrics
- Add correlation ID, token counts, and latency tracking
- Replace console.log with app.logger throughout"

Done-When

• Scope truth table created in SCOPE_AUDIT.md
• Unused scopes removed from manifest
• Bot fails gracefully when accessing removed scopes
• All AI operations log structured data
• Logs queryable by correlation ID in Vercel

Solution

1. Example manifest with trimmed scopes:

{
  "oauth_config": {
    "scopes": {
      "bot": [
        "app_mentions:read",
        "channels:history",
        "chat:write",
        "im:history"
      ]
    }
  }
}

2. Structured logging middleware enhancement:

export const correlationMiddleware: Middleware<SlackEventMiddlewareArgs> = async ({
  context,
  body,
  next,
  logger
}) => {
  const event = 'event' in body ? body.event : null;
  const correlationId = event
    ? `${event.type}_${event.event_id}_${Date.now()}`
    : `req_${Date.now()}`;
 
  const correlation = {
    correlationId,
    event_id: event?.event_id,
    type: event?.type,
    channel: event?.channel,
    ts: event?.ts,
    thread_ts: event?.thread_ts ?? event?.ts,
    user: event?.user
  };
 
  // Attach correlation to context so all loggers can spread it
  // logger.info({ ...context.correlation, ... }, 'message')
  // logger.error({ ...context.correlation, error }, 'message')
  (context as any).correlation = correlation;
 
  await next();
};

3. AI operation logging:

// Inside your existing respondToMessage implementation (with cost control from 4.5)
const startTime = Date.now();
 
// ...existing AI request code (generateText / withRetry / cost tracking)...
 
app.logger.info("AI response completed", {
  ...correlation,
  operation: "respondToMessage",
  // keep any existing cost fields from lesson 4.5 (actualCost, estimatedCost, estimationAccuracy, ...)
  promptTokens: result.usage.inputTokens,
  completionTokens: result.usage.outputTokens,
  channel: event.channel,
  thread_ts: thread_ts || event.ts,
  latencyMs: Date.now() - startTime,
});
 
// In your catch block:
app.logger.error("AI response failed", {
  ...correlation,
  operation: "respondToMessage",
  error: error instanceof Error ? error.message : String(error),
  channel: event.channel,
  thread_ts: thread_ts || event.ts,
  latencyMs: Date.now() - startTime,
});

Key Takeaways

• Every OAuth scope is a security risk - only keep what you actually use
• Structured logging with correlation IDs makes debugging possible at scale
• Token tracking is essential for cost management
• Scope audits should be part of your CI/CD pipeline
• Use app.logger everywhere - console.log is dead to us