Summary

Making a local only ai agent that can complete coding tasks in various language and take projects from start to finish. How would such a system work? What are the required pieces needed to make it function well? How can it be extended to add more capabilities?

Objectives

  1. Create an ai agent that can be run from the terminal.
  2. It should have the ability to code in multiple languages and iterate on simple ideas.
  3. It should have the ability to plan projects and write a project requirements document in order to faithfully execute the intended application.
  4. It should have the ability to point out where errors in logic or poorly defined requirements and expectations have been found.

Future Addition

Voice mode - This will be a full speech to speech style interface that will encapsulate all the functionalities of the agent. This would allow for voice interaction with the agent to plan and execute on any idea at any time. - The voice mode should adjust how output is given. I will give notifications on long running tasks. Check in on progress and generally refrain from reciting every single word and step of a process unless asked. Soft launch or sandbox running of any project created using the agent. THe agent should be able to test if the project is in a runnable state. If it is not it should proactively try to fix any mistakes and run it again, as long as it remains below a max retry limit. Once that is reached it should ask for help.

Why

The purpose of this is not to make the most state of the art agent. It is instead to gain a better understanding of how to make and maintain some of these systems. How to add functionality and how to adequately customize them for multiple situations.

Tech

  • Python
  • Ollama
  • markdown

Concerns

  • Not knowing what I am doing at all
  • System fragility, as iteration takes time even though it is being worked on in concert with an ai
  • So far the first version is pretty awful and simple to contain, but there is a possibility that given access to a computer the option for it to find a way outside of the main project directory and cause harm to the underlying system.