I’ve been trying to set up AI agents as my portfolio trader since last year. Not looking for fancy strategies just good old fashion fundamental investing. But it never got to a point where I’m confident enough to try with real money - even the best model + harness could suddenly spit out a horrible trade due to some dumb logic or hallucination. I just kept on tweaking the setup and manually scanning outputs to see if they are sensible. I'm fine with a bad decision once awhile but I don't know how frequent they will be in real setting when it is screening dozens of stocks over a long period. You can't deploy if you don't understand your AI enough, but if you don't understand your AI enough you can't deploy! You can paper trade but you need a loooonggg period until you can be sure and by that time a better model might be out and u need to test that again...
Then it hit me - there is a scalable way to understand your trading agent - backtesting! Just like how u backtest a signal/algo, u can teleport the agent back in time and let it make decisions. If the environment and data around the agent is controlled, the agent would never know. And you can do this for hundreds of time to understand their aggregate behaviour and performance . A few bad days is fine if, in a long horizon, it generates stable and positive Sharpe/IR.
Well I was kind of naive bc this entails infrastructure that doesn't exist, essentially a coding eval pipeline but catered for financial agents with flexibility to design and own your benchmark based on your strategy...
Long story short, I went through the pain and spent more than 5 months building it. With it, I blasted through variations of my strategy + agent. Now I know that if I had deployed my agent 4 years ago to cover DJIA stocks, its IR today sits at 1.19, max drawdown is -9% (slightly better than DJIA). I know it tends to perform better when DJIA is down (bc it was focusing on fundamental health) but tends to underweight industrials regardless. I also know that it rarely looks beyond 3 years for historical ratios, ignore 8K filings and too sensitive to news of stock rating upgrades from GS/JPM. Moreover, I know what happens if I had used GPT over Claude as the underlying model, or add a risk manager subagent to the mix or an extra prompt to make it more diligent.
With this eval pipeline, I then iteratively improved my agent to the point where, in April this year, I finally handed 100k USD of my savings to let it trade. The performance so far is very aligned to my expectation- currently outperforming its benchmark by 5%.
Coming from the fund industry, I think this is a game changer. Not because I got AI to trade for me but because there is a systematic way to evaluate and improve your AI on making investment decisions. You don't have to fly blind; you can build and run your own institutional-grade AI systematic fund today. I'm sharing everything on my site, from [raw performance and outputs of my AI portfolio](https://www.fintel.capital/strategy), to [how I did it ](https://www.fintel.capital/blogs/iterative-agent-improvement)and the [open-source eval pipeline](https://github.com/felixdaga/fintel) which you can pull and run evals on your own AI strategy and agent today. Hope this can inspire you!
