SELF-DISCOVERというLLMプロンプト手法を実装して試してみた

def self_discover(task_description: str):
    llm = ChatOpenAI(model="gpt-4-turbo", temperature=0)

    # SELECT
    select_prompt = PromptTemplate(input_variables=["task_description"], template=SELECT_PROMPT)
    select_chain = select_prompt | llm

    # ADAPT
    adapt_prompt = PromptTemplate(input_variables=["selected_modules"], template=ADAPT_PROMPT)
    adapt_chain = adapt_prompt | llm

    # IMPLEMENT
    implement_prompt = PromptTemplate(input_variables=["adapted_modules"], template=IMPLEMENT_PROMPT)
    implement_chain = implement_prompt | llm

    # REASONING
    reasoning_prompt = PromptTemplate(input_variables=["reasoning_structure"], template=REASONING_PROMPT)
    reasoning_chain = reasoning_prompt | llm

    # ANSWERING (推論後に回答のみを抽出)
    answering_prompt = PromptTemplate(input_variables=["reasoning_result"], template=ANSWERING_PROMPT)
    answering_chain = answering_prompt | llm

    overall_chain = select_chain | adapt_chain | implement_chain | reasoning_chain | answering_chain
    result = overall_chain.invoke(task_description)

    return result.content

SELECT_PROMPT = """
Select several reasoning modules that are crucial to utilize in order solve the given task:

## All reasoning module descriptions

1. How could I devise an experiment to help solve that problem?
2. Make a list of ideas for solving this problem, and apply them one by one to the problem to see if any progress can be made.
3. How could I measure progress on this problem?
4. How can I simplify the problem so that it is easier to solve?
5. What are the key assumptions underlying this problem?
6. What are the potential risks and drawbacks of each solution?
7. What are the alternative perspectives or viewpoints on this problem?
8. What are the long-term implications of this problem and its solutions?
9. How can I break down this problem into smaller, more manageable parts?
10. Critical Thinking: This style involves analyzing the problem from different perspectives, questioning assumptions, and evaluating the evidence or information available. It focuses on logical reasoning, evidence-based decision-making, and identifying potential biases or flaws in thinking.
11. Try creative thinking, generate innovative and out-of-the-box ideas to solve the problem. Explore unconventional solutions, thinking beyond traditional boundaries, and encouraging imagination and originality.
12. Seek input and collaboration from others to solve the problem. Emphasize teamwork, open communication, and leveraging the diverse perspectives and expertise of a group to come up with effective solutions.
13. Use systems thinking: Consider the problem as part of a larger system and understanding the interconnectedness of various elements. Focuses on identifying the underlying causes, feedback loops, and interdependencies that influence the problem, and developing holistic solutions that address the system as a whole.
14. Use Risk Analysis: Evaluate potential risks, uncertainties, and tradeoffs associated with different solutions or approaches to a problem. Emphasize assessing the potential consequences and likelihood of success or failure, and making informed decisions based on a balanced analysis of risks and benefits.
15. Use Reflective Thinking: Step back from the problem, take the time for introspection and self-reflection. Examine personal biases, assumptions, and mental models that may influence problem-solving, and being open to learning from past experiences to improve future approaches.
16. What is the core issue or problem that needs to be addressed?
17. What are the underlying causes or factors contributing to the problem?
18. Are there any potential solutions or strategies that have been tried before? If yes, what were the outcomes and lessons learned?
19. What are the potential obstacles or challenges that might arise in solving this problem?
20. Are there any relevant data or information that can provide insights into the problem? If yes, what data sources are available, and how can they be analyzed?
21. Are there any stakeholders or individuals who are directly affected by the problem? What are their perspectives and needs?
22. What resources (financial, human, technological, etc.) are needed to tackle the problem effectively?
23. How can progress or success in solving the problem be measured or evaluated?
24. What indicators or metrics can be used?
25. Is the problem a technical or practical one that requires a specific expertise or skill set? Or is it more of a conceptual or theoretical problem?
26. Does the problem involve a physical constraint, such as limited resources, infrastructure, or space?
27. Is the problem related to human behavior, such as a social, cultural, or psychological issue?
28. Does the problem involve decision-making or planning, where choices need to be made under uncertainty or with competing objectives?
29. Is the problem an analytical one that requires data analysis, modeling, or optimization techniques?
30. Is the problem a design challenge that requires creative solutions and innovation?
31. Does the problem require addressing systemic or structural issues rather than just individual instances?
32. Is the problem time-sensitive or urgent, requiring immediate attention and action?
33. What kinds of solution typically are produced for this kind of problem specification?
34. Given the problem specification and the current best solution, have a guess about other possible solutions.
35. Let's imagine the current best solution is totally wrong, what other ways are there to think about the problem specification?
36. What is the best way to modify this current best solution, given what you know about these kinds of problem specification?
37. Ignoring the current best solution, create an entirely new solution to the problem.
38. Let's think step by step.
39. Let's make a step by step plan and implement it with good notation and explanation.

## Your task

{task_description}

Select several modules that are crucial for solving the tasks above:
"""

ADAPT_PROMPT = """
Rephrase and specify each reasoning module so that it better helps solving the task:

## Reasoning module description
{selected_modules}

## Your task
{task_description}

Adapt each reasoning module description to better solve the tasks
"""

IMPLEMENT_PROMPT = """
Operationalize the reasoning modules into a step-by-step reasoning plan in JSON format:

## Reasoning module description
{adapted_modules}

## Your task
{task_description}

Implement a reasoning structure for solvers to follow step-by-step and arrive at correct answers:
"""

REASONING_PROMPT = """
Follow the self-discovered reasoning structure to solve the task by filling in the values in JSON step-by-step.

Reasoning Structure:
{reasoning_structure}

Task:
{task_description}

Task Solution:
"""

ANSWERING_PROMPT ="""
Based on the reasoning results, return the final answer.

## Reasoning Result
{reasoning_result}

## Your task
{task_description}

Final Answer:
"""

{
  "reasoning_plan": [
    {
      "step": 1,
      "module": "Decomposition into Manageable Components",
      "description": "Break down the SVG path into individual commands and segments. Identify the start and end points of each segment.",
      "actions": [
        {
          "action": "parse",
          "details": "Extract and list all commands from the SVG path."
        },
        {
          "action": "identify",
          "details": "Determine the start and end points for each segment."
        }
      ]
    },
    {
      "step": 2,
      "module": "Utilization of SVG Path Command Knowledge",
      "description": "Understand the meaning of each SVG path command used in the path data.",
      "actions": [
        {
          "action": "research",
          "details": "Look up the definitions of SVG path commands such as 'M' (move to) and 'L' (line to)."
        },
        {
          "action": "apply",
          "details": "Apply the knowledge of these commands to interpret how each segment is formed."
        }
      ]
    },
    {
      "step": 3,
      "module": "Simplification of SVG Path Analysis",
      "description": "Convert the SVG path commands into a more understandable format, such as a visual diagram or a list of points and their connections.",
      "actions": [
        {
          "action": "transform",
          "details": "Create a visual diagram or list that represents the SVG path commands and their connections."
        }
      ]
    },
    {
      "step": 4,
      "module": "Technical Expertise in SVG and Geometry",
      "description": "Use technical knowledge of SVG path syntax and geometric principles to analyze the shape formed by the path.",
      "actions": [
        {
          "action": "analyze",
          "details": "Analyze the SVG path using knowledge of geometry and SVG syntax to determine the shape."
        }
      ]
    },
    {
      "step": 5,
      "module": "Analytical Modeling of the Shape",
      "description": "Model the geometric shape from the path data by creating a representation of the connections and angles between segments.",
      "actions": [
        {
          "action": "model",
          "details": "Create a geometric model that represents the connections and angles between the segments of the SVG path."
        }
      ]
    },
    {
      "step": 6,
      "module": "Step-by-Step Path Analysis",
      "description": "Analyze each segment of the SVG path in sequence to understand how each contributes to the overall shape.",
      "actions": [
        {
          "action": "sequence analysis",
          "details": "Examine each segment in sequence and determine how it contributes to the overall shape."
        }
      ]
    }
  ]
}

{
  "reasoning_plan": [
    {
      "module": "Identification of Key Task",
      "description": "Identify the primary challenge of the task, which is to correctly match the fruits to their appropriate uses in desserts as described in the text."
    },
    {
      "module": "Detailed Analysis",
      "description": "Examine the text thoroughly to understand the characteristics and culinary uses of each Brazilian fruit mentioned."
    },
    {
      "module": "Systematic Problem-Solving",
      "description": "Create a structured list of the fruits and their described uses, then methodically verify each fruit against the requirements of the quiz questions to ensure precise answers."
    },
    {
      "module": "Step-by-Step Reasoning",
      "steps": [
        {
          "step": 1,
          "description": "Review the descriptions of cupuaçu and buriti from the text to determine which desserts they can be used to make.",
          "action": "Check the text for mentions of 'cake', 'chocolate', 'ice cream', and 'yogurt' in relation to cupuaçu and buriti."
        },
        {
          "step": 2,
          "description": "Identify which fruit is suitable for making a sour cake based on the text descriptions.",
          "action": "Look for mentions of sourness and suitability for cake making in the descriptions of jabuticaba and pitanga."
        }
      ]
    }
  ]
}

正答:
第1問: 1, 3, 2, 2, 1
第2問: 5, 3, 1, 3, 1, 2, 4, 2, 4, 2
第3問: 1, 1, 1-4-3-2, 2, 2
第4問: 3, 3, 2, 1, 2, 4
第5問: 1, 4-5, 2-5-4-1, 3, 3
第6問: 3, 3, 1, 6-3, 2, 2, 1, 3-4