Rendered at 21:33:47 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
alecbz 2 hours ago [-]
We have LLMs try to generate descriptions of PRs for us and they're pretty universally disliked. They're always overly-complex descriptions of the mechanical changes and have no sense of motivation.
Also, a huge reason to understand the code yourself is to make sure the LLM isn't wrong, but this doesn't work if an LLM is itself generating the understanding.
storus 43 minutes ago [-]
My main gripe is with Claude deciding to make 200 lines of code in a PR I need to review, instead of 3 lines of code somebody who understands the original algorithm/intent would do. And coworkers just YOLOing changes without understanding them. Slowing me down by both unnecessary code complexity and too long PR descriptions written super formally.
3abiton 2 minutes ago [-]
I hate to be pedantic but you can finetune a skill to shape the PR message the way you like it. That being said, I did have exactly this issue you mentioned, but the defualt output can always be tuned.
dylan604 1 hours ago [-]
Are these generated descriptions of LLM submitted changes or of human changes? If a human, shouldn't they be putting the motivation into the PR?
alecbz 1 hours ago [-]
LLM-generated (or at least LLM-assisted), but a human's still the one submitting the PR for review.
dylan604 1 hours ago [-]
A PR with a minimal title and empty description should be refused at submission. If the human is so disinterested that they're using LLM generated code and then can't explain the purpose, that human should be prevent from making the PR. Working as a solo dev, it is very easy to be lazy like that, and I'm as guilty as anyone. Working in teams with actual reviews should absolutely have much more strict policies of what is considered a valid PR
alecbz 58 minutes ago [-]
I mean, the “meta” right now is trying to automate as much of the software engineering flow as possible, including writing PR descriptions.
They’re shooting for LLMs being able to one-shot PRs or need minimal oversight. But yeah, in practice LLMs are not there IME.
dylan604 20 minutes ago [-]
This is the biggest issue I have with current state of affairs. It's not there yet. Because of that, extra work is needed to get them to work that otherwise would not need to be spent. Everyone is shouting from the roof tops about how great things are while suppressing these types of issues.
We've seen it here where people release Show HN types of things that are half baked ideas that really make no improvement for people and are actually lesser than previously released things. Yet they are expecting people to be amazed. Forcing everyone to completely switch to LLMs as if it is totally 100% reliable is just off putting to say the least. It takes discussing things with people honestly looking at the situation to have any semblance of thinking you're not the insane one for pushing back
vjvjvjvjghv 2 hours ago [-]
I don’t know. From my experience I get pretty good descriptions of PRs if I ask the right questions and provide some context.
jollyllama 54 minutes ago [-]
> provide some context
That's pretty much what a PR description is.
nsingh2 15 minutes ago [-]
Work doesn't start with a PR description though. I'm assuming most people that are using LLMs start with some sort of document (plan, spec, intent, etc) which captures intent.
I guess you could also use all the session rollouts saved to disk that were related to that task, and distill them somehow.
baq 35 minutes ago [-]
The difference is an LLM can convert a stream of consciousness into well-formed prose for approximately free; I assume ‘provide some context’ means ‘brain dump’ in the OP
alecbz 9 minutes ago [-]
This has unfortunately not been my experience at all. Often LLMs miss or get wrong subtle details when I don't do the pre-work to organize my thoughts well ahead of time (at which point it's unclear how much value they're providing).
22 minutes ago [-]
morkalork 2 hours ago [-]
I am so very tired of 2 page long PR descriptions for a 5 line change.
baq 34 minutes ago [-]
Some 5-line changes deserve a phd.
But yeah, most probably don’t.
avaer 1 hours ago [-]
Maybe on your team, but I don't think AI PRs are universally disliked. The people that submit PRs without understanding them are universally disliked.
Have you tried writing in AGENTS.md or whatever to exactly explain what you like/dislike about the PR descriptions?
alecbz 51 minutes ago [-]
The PR descriptions are pretty universally disliked. We have centralized tooling that manages the prompts for that, I’m sure they’ve tried tuning it but maybe there’s more they could do.
Though I have some local workflows where I try to teach Claude about my writing style preferences via skills and examples, and it’s still not great.
thombles 44 minutes ago [-]
It’s definitely possible to get much better output with prompting. I know, because when I’m faced with a “standard” PR description full of technical clutter, I can paste the link to Claude and ask “ELI5 what the problem actually is, any important context, what changed, and why that solves the problem.” And most of the time it converts it into something pretty good and readable.
bckr 55 minutes ago [-]
The basics are always so basic yet so necessary. Thanks for the recommendation.
w10-1 1 hours ago [-]
I agree with the problem but not the solutions.
The problem pre-dates LLM's: writing code that "works" but breaks the underlying model. Because it works, it always sounds reasonable and doesn't raise any flags.
Only someone - human or LLM - who holds the model as the standard would see that this working solution breaks the model.
(In theory, the model is to preserve scaling, flexibility or some other systemic feature not immediately invalidated by this working code, but as always the model itself could be bad.)
LLM's are not bad at giving an account of the model; indeed, fighting with the LLM over what the model is can clarify things. But LLM's will happily hold on to a stream of inconsistent statements as their model, so they are not the authority.
cyanydeez 1 hours ago [-]
understanding a different modality of model interaction gave me proper insight into the specific problem. In visual models, even if the model understands the concept of face, or hand, or whatever, it doesn't know how to de-dupe a statement like "count the number of faces" until you give it a countable reference frame, so it can internally, place a box around a face and give that a coordinate, and then it can collect all the coordinates, and suddenly it's counting face in a picture.
The same thing happens in code. Things we're happily shifting from context to context, the model itself isn't doing. When it reads file1 for the main() clause, it will easily read file2's main() clause as the same. It'll internally merge these.
So if you do want to work with these models to achieve complex tasks, you basically do have to go reverse centaur and bend the code base to it's blindness. You can't use the same function names across the code base; each one needs to be dstinguishable; same thing with variables that represent seperate entity relationships.
You do that, and it suddenly because a whole lot smarter.
sajithdilshan 3 minutes ago [-]
Understanding has always been the bottleneck. Sometimes AI helps with it like explaining things pretty well with diagrams. However, in general I agree that more code is being generated per developer and it's difficult to keep up with the phase of new changes and understand it.
iainctduncan 2 hours ago [-]
I am so dying to read more about the new/current/real bottleneck!
Where is the bottleneck? WHERE?? Tell me! No evidence needed, just lay it on, man to man, thought-leader to thought-leader!
euroderf 1 hours ago [-]
> thought-leader to thought-leader!
This is my new chat-up line at networking events.
techpression 21 minutes ago [-]
It doesn’t matter, once you found the bottleneck there is a new one. Seems we changed the supposed bottleneck of writing code (as if it ever were, the world was producing far too much code before LLMs were even a thing) with about ten or so new ones, was it a good trade?
iainctduncan 2 hours ago [-]
Oh wait, there it is, sitting over top of the fat part and under the cork...
euthymiclabs 3 hours ago [-]
"I read the code." -Mitchell Hashimoto
Great code needs great understanding and agents need excellent guidance. Even in my current solo-dev work, I can't imagine making a production commit I haven't read until I understand it. I own the consequences of my code; that's a responsibility AI agents can't take.
a2ff6eeb0 27 minutes ago [-]
Understanding was always the bottleneck. The way LLMs speed up your work is by letting you get code without taking the time to understand it. If you want to understand your code, LLMs are a net loss.
If you want to move faster with LLMs, you need to act like a manager and stop caring about what the LLM did. You just need to do the manual testing and make sure it works.
hk__2 2 hours ago [-]
For me the solution has been to throw away the code I don’t understand. I let the agent write the code, and if when I read it it seems unclear or needs a lot of explanation from the agent, I just throw it away and start over, or do it by myself.
While the tips are good to handle the volume, I still think this sets code owner on a dangerous path.
AI have limitation and hallucinate. Complex code will be explained in hallucinated way. At some point AI will be unable to write more because the arch has become too complex or the volume of code will be to high.
The article I would like to read would suggest how to force LLM to architect the code like a solid tower instead of a pile of unstable mud.
vjvjvjvjghv 2 hours ago [-]
I feel if you still architect the code and guide the LLM, it will do a pretty good job. Maybe one day the LLM will be able to do all the system architecture but that’s probably still quite some time out. I don’t even know if that’s possible considering different business needs and other factors that aren’t technical.
alikhater30000 1 hours ago [-]
[flagged]
the_arun 23 minutes ago [-]
Understanding is always a bottleneck regardless of human or ai. But now we are at a different scale.
dtkav 56 minutes ago [-]
I've been using Geoffrey's /explain-diff skill in my replace-github-with-tailor-fit-personal-software journey, and I'm liking it. I recommend at least giving it a try.
threethirtytwo 7 minutes ago [-]
This is a temporary bottleneck. AI is moving so fast that this will change. Wait six months and this article is no longer relevant.
About a year ago most people were still typing code. Having an agent do ALL code was crazy.
Within a year or two years at most, a lot of people will stop trying to understand code. The onus will shift to testing and QAing.
wseqyrku 1 hours ago [-]
If you try to spec the problem with all the painful details for the machine to understand, you will end up with a rust codebase.
bigstrat2003 1 hours ago [-]
Understanding has always been the bottleneck. That's why LLMs aren't actually helpful: they speed up the part which is easy (typing characters into your editor), but are neutral or even harmful on the part which is hard (understanding the problem and how best to solve it).
causal 32 minutes ago [-]
It seems like humans have a limited "understanding budget" but LLMs force us to spend that understanding on waaaaay more code and projects than ever before.
stronglikedan 1 hours ago [-]
The part which is easy is still time consuming, so LLMs are helpful. They're just not a silver bullet.
elendilm 1 hours ago [-]
Understanding is expensive. And hence valuable.
LLMs usually points to the most idiotic future trajectory on my work, and I have to curse it inorder to let it keep up with my refined understanding.
But what else would one expect from a probabilistic weighted next token predictor, other than to conduct probabilistic search which are 99.99% deadends.
But LLMs can pave the way towards constructing resilient and correct architecture which can be iterated fast by a human.
Architecture and determinism is where my money is in.
jbdamask 2 hours ago [-]
pfft...I'm way past understanding
ch4s3 2 hours ago [-]
Overstanding is the new horizon
stronglikedan 1 hours ago [-]
That's already been co-opted by the sovereign citizen movement, so use with caution lest you be judged for it.
Also, a huge reason to understand the code yourself is to make sure the LLM isn't wrong, but this doesn't work if an LLM is itself generating the understanding.
They’re shooting for LLMs being able to one-shot PRs or need minimal oversight. But yeah, in practice LLMs are not there IME.
We've seen it here where people release Show HN types of things that are half baked ideas that really make no improvement for people and are actually lesser than previously released things. Yet they are expecting people to be amazed. Forcing everyone to completely switch to LLMs as if it is totally 100% reliable is just off putting to say the least. It takes discussing things with people honestly looking at the situation to have any semblance of thinking you're not the insane one for pushing back
That's pretty much what a PR description is.
I guess you could also use all the session rollouts saved to disk that were related to that task, and distill them somehow.
But yeah, most probably don’t.
Have you tried writing in AGENTS.md or whatever to exactly explain what you like/dislike about the PR descriptions?
Though I have some local workflows where I try to teach Claude about my writing style preferences via skills and examples, and it’s still not great.
The problem pre-dates LLM's: writing code that "works" but breaks the underlying model. Because it works, it always sounds reasonable and doesn't raise any flags.
Only someone - human or LLM - who holds the model as the standard would see that this working solution breaks the model.
(In theory, the model is to preserve scaling, flexibility or some other systemic feature not immediately invalidated by this working code, but as always the model itself could be bad.)
LLM's are not bad at giving an account of the model; indeed, fighting with the LLM over what the model is can clarify things. But LLM's will happily hold on to a stream of inconsistent statements as their model, so they are not the authority.
The same thing happens in code. Things we're happily shifting from context to context, the model itself isn't doing. When it reads file1 for the main() clause, it will easily read file2's main() clause as the same. It'll internally merge these.
So if you do want to work with these models to achieve complex tasks, you basically do have to go reverse centaur and bend the code base to it's blindness. You can't use the same function names across the code base; each one needs to be dstinguishable; same thing with variables that represent seperate entity relationships.
You do that, and it suddenly because a whole lot smarter.
Where is the bottleneck? WHERE?? Tell me! No evidence needed, just lay it on, man to man, thought-leader to thought-leader!
This is my new chat-up line at networking events.
Great code needs great understanding and agents need excellent guidance. Even in my current solo-dev work, I can't imagine making a production commit I haven't read until I understand it. I own the consequences of my code; that's a responsibility AI agents can't take.
If you want to move faster with LLMs, you need to act like a manager and stop caring about what the LLM did. You just need to do the manual testing and make sure it works.
AI have limitation and hallucinate. Complex code will be explained in hallucinated way. At some point AI will be unable to write more because the arch has become too complex or the volume of code will be to high.
The article I would like to read would suggest how to force LLM to architect the code like a solid tower instead of a pile of unstable mud.
About a year ago most people were still typing code. Having an agent do ALL code was crazy.
Within a year or two years at most, a lot of people will stop trying to understand code. The onus will shift to testing and QAing.
LLMs usually points to the most idiotic future trajectory on my work, and I have to curse it inorder to let it keep up with my refined understanding.
But what else would one expect from a probabilistic weighted next token predictor, other than to conduct probabilistic search which are 99.99% deadends.
But LLMs can pave the way towards constructing resilient and correct architecture which can be iterated fast by a human.
Architecture and determinism is where my money is in.