Skip to content

Quiz: AI Tools and Strategy for Technical PMs

Test your understanding of generative AI, prompt engineering, AI limitations, and strategic AI integration with these review questions.

1. How do large language models generate text?

  1. By predicting the most likely next token given all preceding tokens based on statistical patterns
  2. By querying a structured database of pre-written responses and selecting the best match
  3. By applying a set of explicitly programmed grammar rules to construct grammatically correct sentences
  4. By searching the internet in real time and assembling relevant information into a coherent response
Show Answer

The correct answer is A. The chapter explains that large language models work by "predicting the most likely next token (word or word fragment) given all preceding tokens," describing this as "a deceptively simple mechanism that produces sophisticated reasoning, creative writing, and code generation." The chapter also warns that LLMs "do not think - they predict," emphasizing that their outputs come from statistical prediction rather than reasoning from first principles. Understanding this mechanism helps PMs set appropriate expectations with stakeholders.

Concept Tested: Large Language Models

2. What distinguishes Claude from ChatGPT as a tool for product managers according to the chapter?

  1. Claude can generate images while ChatGPT is limited to text-only outputs
  2. Claude offers stronger long-document analysis and nuanced reasoning about trade-offs and edge cases
  3. Claude is exclusively designed for code generation while ChatGPT handles general writing tasks
  4. Claude provides real-time internet search capabilities that ChatGPT does not support
Show Answer

The correct answer is B. The chapter describes Claude as "particularly strong at nuanced analysis, following complex instructions, and working with long documents," noting that its extended context window makes it "especially useful for analyzing entire specifications, codebases, or research corpuses in a single conversation." Claude's distinctive strengths are listed as analyzing long documents, nuanced reasoning about trade-offs, following detailed multi-step instructions, and careful handling of ambiguous requirements. ChatGPT is described as excelling at drafting, brainstorming, and summarization tasks.

Concept Tested: Claude for PMs

3. A PM needs to get an AI tool to analyze a competitor's pricing strategy. Which prompt engineering principle should they apply first to improve output quality?

  1. Keep the prompt as short as possible to reduce token consumption and model confusion
  2. Ask the AI to generate multiple responses and manually select the best one
  3. Provide specific context including the product type, target market, and desired output format
  4. Use technical jargon to signal expertise and trigger more sophisticated model responses
Show Answer

The correct answer is C. The chapter's prompt engineering section emphasizes that being specific and providing context are foundational principles. The prompt quality comparison table shows that vague prompts like "Help me with my product" produce generic results, while prompts including specific context such as product type, data, and desired format produce "specific, actionable analysis." The five core principles listed are: be specific, provide context, define the output format, assign a role, and iterate and refine. Adding context transforms a weak prompt into one that produces useful output.

Concept Tested: AI Prompt Engineering

4. Which of the following is described in the chapter as one of the highest-value AI applications for PMs transitioning from non-technical backgrounds?

  1. Using AI to automatically deploy code changes to production environments
  2. Using AI to read, explain, and analyze source code without requiring deep programming expertise
  3. Using AI to replace the need for engineering team members on small projects
  4. Using AI to generate final production-ready code that requires no human review
Show Answer

The correct answer is B. The chapter states that AI code understanding "is one of the highest-value applications of AI" for PMs transitioning from non-technical backgrounds. It explains that PMs can "paste a code snippet, a pull request diff, or an error log into an AI tool and ask it to explain what the code does, identify potential issues, or suggest improvements." Practical applications include pull request review, architecture comprehension, error interpretation, SQL query review, and API contract analysis. The emphasis is on understanding code, not replacing engineers or deploying code.

Concept Tested: AI Code Understanding

5. A PM is proposing an AI-powered recommendation feature. According to the chapter's strategic framework, which question should be evaluated first?

  1. Which competitor has already launched a similar AI-powered feature
  2. How many engineers are available to build the recommendation engine
  3. What AI vendor offers the lowest per-token pricing for the recommendation model
  4. What specific user pain point does AI solve better than non-AI alternatives
Show Answer

The correct answer is D. The chapter presents a six-step strategic framework for evaluating AI opportunities, and the first question is "User pain point - What specific user problem does AI solve better than alternatives?" The chapter emphasizes that the strategic question is not "how do we add AI?" but "where does AI create meaningful value for our users that justifies the cost and complexity?" Starting with the user pain point ensures that AI adoption is driven by genuine value creation rather than technology enthusiasm or competitive pressure.

Concept Tested: AI in Product Strategy

6. A technical PM is comparing synchronous and asynchronous AI integration patterns for a document analysis feature. What is the key trade-off between these two patterns?

  1. Synchronous patterns are cheaper to operate while asynchronous patterns require more expensive infrastructure
  2. Synchronous patterns require the user to wait for a response while asynchronous patterns can handle longer processing in the background
  3. Synchronous patterns only work with text data while asynchronous patterns support multimodal inputs
  4. Synchronous patterns are more secure because data never leaves the main application server
Show Answer

The correct answer is B. The chapter's AI integration architecture section contrasts the two patterns: synchronous processing has "200ms-5s total latency" where the "user waits" and is best for "chat interfaces, real-time suggestions, code completion," while asynchronous processing takes "seconds to minutes" with "lower urgency" and is "better for complex tasks" like "document analysis, batch processing, content generation." For a document analysis feature, asynchronous processing would likely be more appropriate since it can handle longer processing times without the user waiting.

Concept Tested: AI Integration Planning

  1. Increase the model's temperature setting to generate more diverse and accurate outputs
  2. Implement retrieval-augmented generation and real-time data feeds
  3. Apply human review, fact-checking, and confidence scores to AI-generated content
  4. Restrict the AI model to only respond with information from its training data
Show Answer

The correct answer is C. The chapter's limitations and mitigation table explicitly lists the mitigation for hallucination as "Human review, fact-checking, confidence scores." The chapter distinguishes this from the mitigation for knowledge cutoff, which is "RAG (retrieval-augmented generation), real-time data feeds" (option B). The chapter emphasizes that hallucination "is not a rare edge case; it is an inherent property of probabilistic text generation," making human oversight essential rather than optional for user-facing applications.

Concept Tested: AI Limitations

8. An AI prototype that costs $50 per month for 100 test users is being considered for production launch targeting 100,000 users. What does the chapter warn about in this scenario?

  1. AI API costs can scale non-linearly, potentially reaching $50,000 per month at production scale
  2. The prototype will require a complete rewrite because AI prototypes never scale to production
  3. The vendor will likely increase pricing once they detect production-level usage patterns
  4. The engineering team will need to double in size to support the increased user base
Show Answer

The correct answer is A. The chapter includes a specific warning box titled "AI Cost Surprises" that states: "A prototype that costs $50/month for 100 test users might cost $50,000/month at 100,000 users." It advises PMs to "always model costs at target scale before committing to an AI-powered feature, and build cost monitoring into your integration from day one." This illustrates the broader principle from the AI cost-benefit analysis section that AI costs can scale non-linearly, making financial planning essential before production commitments.

Concept Tested: AI Cost-Benefit Analysis

9. Which component of an AI governance framework is responsible for categorizing AI features by their potential for harm?

  1. AI use policy that defines approved tools and acceptable use cases
  2. Audit trail that documents AI decisions and model versions
  3. Incident response procedures for handling AI failures and data breaches
  4. Risk classification system that categorizes AI features by risk level
Show Answer

The correct answer is D. The chapter lists six components of an effective AI governance framework, including "Risk classification - A system for categorizing AI features by risk level" with examples ranging from "low risk: content summarization" to "high risk: automated credit decisions." This classification determines the level of scrutiny and review required before deploying an AI feature. While AI use policy, audit trails, and incident response are all important governance components, risk classification is specifically responsible for categorizing features by their potential for harm.

Concept Tested: AI Governance

10. If you were asked to design an AI governance framework for a mid-size SaaS company launching its first AI-powered feature, which element would you prioritize implementing first and why?

  1. A comprehensive audit trail system to document every AI decision before any feature launches
  2. An annual assessment process to review AI systems for continued accuracy and fairness
  3. A risk classification system paired with a review process for high-risk applications
  4. An incident response plan that covers every possible AI failure mode across all product lines
Show Answer

The correct answer is C. For a company launching its first AI feature, establishing a risk classification system paired with a review process creates the foundation for responsible AI deployment without over-engineering governance for a single feature. The chapter defines risk classification as categorizing features by risk level (low to high) and mandates review for "high-risk AI applications, involving legal, ethics, and technical stakeholders." Starting here ensures the first feature launches responsibly while creating a scalable framework. Audit trails and annual assessments are important but less urgent for a first launch, and incident response for all product lines is premature for a company with no AI features yet in production.

Concept Tested: AI Governance