Harness Engineering Follow Up: Our Mini (Unscientific) Experiment
A couple weeks ago we wrote this blog post about harness engineering. The gist is that the governance around an AI model is becoming more important than the model itself (please read the full post for details). What an AI agent can do, and how it can do it, can open up new attack surfaces for bad actors in the worst case, and at least needs to be optimized in the best case.
Our Question
A lot of our work takes place in high compliance environments. As such, we tend to look deeper into processes and settings than teams operating outside of compliance mandates. Harness engineering is getting more and more attention, but we wondered if that was common across the board, or was it just in our world of extra security scrutiny? Are smaller teams in the private sector operating so fast that they aren’t thinking about how their tools are connected? Are developers plugging in tools themselves, instead of having a corporate strategy, and only focusing on improving their own output?
We decided to find out with a simple experiment.
The Experiment
To get a gauge on how the market sees harness engineering we turned to Reddit. Yes, we know there are flaws in the method, but the title of this blog says unscientific after all. Anyway, we posted a question about the attention being paid to harness engineering in both the regular DevOps subreddit and the DevSecOps subreddit (links to both posts are at the bottom of this blog). The expectation was that teams with specific security members would be paying more attention to harness engineering, as well as larger teams in general.
The Results
As you may have already guessed, we were largely correct in our hypothesis. As of writing this, the post in r/DevSecOps has ~2,900 views, 7 upvotes, and only 6 comments. The post in r/DevOps has ~36,000 views, 20 upvotes, and 19 comments. The security focused community is smaller, and the responses were as expected with the first comment being “Tons, it’s the new AI frontier.” In r/DevOps however there was a much wider mix. One commenter questioned why anyone would write their own harness when they could just use someone else’s, others talked about minimum usefulness and just using whatever ships with your chosen tool. Another commentor did make a point about differentiating between a harness that compensates for model weaknesses and one that is tied in with policy. Governing how an AI agent can function is an important optimization in harness engineering.
Our Conclusions
Overall, we were not surprised that the more security conscious teams were already paying attention to their harnesses. Knowing though that this is a smaller subset of DevOps professionals in general, it makes us wonder about issues hiding in pipelines beyond the industry’s typical CVE attention. In the best case, an AI agent could ship a broken feature or fix a bug that clearly creates another one. In the worst, they could expose proprietary information or access keys in a public repo. These situations might not all be breaches, but they are headaches for the developers that could be avoided. They might also go unnoticed for a long time because, depending on your harness, the agents might be acting perfectly as configured.
r/DevSecOps – https://www.reddit.com/r/devsecops/comments/1vfkekb/how_much_attention_is_harness_engineering_getting/
r/DevOps – https://www.reddit.com/r/devops/comments/1vfkoi4/how_much_attention_is_harness_engineering_getting/