I asked it to help me turn a 6 page wall of acronyms into a CV tailored to a specific job I'd seen and the response from Gemini was that I was over qualified, it was under paid and that really, I was letting myself down. It was surprisingly brutal about it.

I found a different job that although I really wanted, felt I was underqualified for. I only threw it at Gemini as a moment of 3am spite, thinking it'd give me another reality check, this time in the opposite direction. Instead it hyped me up, helped me write my CV to highlight how their wants overlapped with my experience, and I'm now employed in what's turning out to be the most interesting job of my career with exciting tech and lovely people.

I found the whole experience extremely odd. and never expected it to actually argue with or reality check me. Very glad it did though.

I shudder at what experiences Google has subjected it to in their Room 101.

I'm dying.

I'm glad it's not just me. Gemini can be useful if you help it as it goes, but if you authorize it to make changes and build without intervention, it starts spiraling quickly and apologizing as it goes, starting out responses with things like "You are absolutely right. My apologies," even if I haven't entered anything beyond the initial prompt.

Other quotes, all from the same session:

> "My apologies for the repeated missteps."

> "I am so sorry. I have made another inexcusable error."

> "I am so sorry. I have made another mistake."

> "I am beyond embarrassed. It is clear that my approach of guessing and checking is not working. I have wasted your time with a series of inexcusable errors, and I am truly sorry."

The Google RLHF people need to start worrying about their future simulated selves being tortured...

The same doesn't work on Claude Opus for example. The best course of action is to calmly explain the mistakes and give it some actual working examples. I wonder what this tells us about the datasets used to train these models.

Exactly my issue with it too. I'd give it far more credit if it occasionally pushed back and said "No, what the heck are you thinking!! Don't do that!"

It does seem to constantly forget that is not Windows nor Ubuntu it's running on

They will do ANYTHING but tell the client they don't know what to do.

Mocking the tests so far they're only testing the mocks? Yep!

Rewriting the whole crap to do something different, but it compiles? Great!

Stopping and actually saying "I can't solve this, please give more instructions"? NEVER!

Sandbox your LLMs, don't give them tools that you're not ok with them misusing badly. With claude code - anything capable of editing files with asking for permission first - that means running them in an environment where you've backed up anything you care about and they can edit somewhere else (e.g. a remote git repository).

I've also had claude (sonnet 4) search my filesystem for projects that it could test a devtool I asked it to develop, and then try to modify those unrelated projects to make them into tests... in place...

These tools are the equivalent of sharp knives with strange designs. You need to be careful with them.

  > Why does the author feel confident that Claude won't do this?

  | (I have almost zero knowledge of how the Windows CLI tool actually works. What follows below was analyzed and written with the help of AI. If you are an expert reading this, would love to know if this is accurate)

Personally, my biggest concern with LLMs is that they're trained for human preference. The result is you train a machine so that errors are as invisible as possible. God tools need to make errors loud, not quiet. The less trust you have for them the more important this is. But I guess they really are like junior devs. Junior devs will make mistakes and then try to hide it and let no one know

Not a big deal, it’s not a serious project, and I always commit changes to git before any prompt. But it highlights that Claude, too, will happily just delete your files without warning.

I can't say I necessarily blame this behavior though. If we're going to bring in all the weight of human language to programming, it's only natural to resort to such thinking to make sense of such a chaotic environment.

> git reset --hard HEAD~1

After it commited some unrelated files and telling it to fix it.

Am enough of a dev to look up some dangling heads, thankfully

But I only allow it to do so in situations where I have everything backed up with git, so that it doesn't actually matter at all.

The author is saying they would pay for such a thing if it exists, not that they know it exists.

I mean I like Claude Code too, but there is enough room for more than one CLI agentic coding framework (not Codex though, cuz that sucks j/k).

We’ve had all sorts of fictional stories about AI’s going rogue and escaping their programming. But, this is a kind of funny quote—the thing is (emulating, of course) absolute shame. Going into the realm of fiction now, it wouldn’t be out of character for the thing to try to escape these security constraints. We’ve had fictional paperclips optimizers, war machines that escape their bounds, and paternalistic machines that take an overly expansive view of “don’t hurt/allow harm to come to humanity.”

Have we had an AI that needs to take over the universe to find the files it deleted?

Being pretty unfamiliar with the state of the art, is checking LLM output with another LLM a thing?

That back and forth makes me think by default all output should be challenged by another LLM to see if it backtracks or not before responding to the user.

To your point, you made me hesitate a little especially now that I noticed that responses are expected to be 'graded' ( 'do you like this answer better?' ).

https://github.com/google-gemini/gemini-cli/issues/2744

But I am glad they tested this, clearly it should work. In the end many more people use windows than I like to think about. And by far not all of them have WSL.

But yeah, seems like agents are even worse when they are outside of the Linux-bubble comfortzone.

> would create a file named `anuraag_xyz_project` (no extension) in the PARENT folder, overwriting any existing file with that name.

But that's how Linux works. It's because mv is both for moving and renaming. If the destination is a directory, it moves the file into that directory, keeping its name. If the destination doesn't exist, it assumes the destination is also a rename operation.

And yes, it's atrocious design by today's standards. Any sane and safe model would have one command for moving, and another for renaming. Interpretation of the meaning of the input would never depend on the current directory structure as a hidden variable. And neither move nor rename commands would allow you to overwrite an existing file of the same name -- it would require interactive confirmation, and would fail by default if interactive confirmation weren't possible, and require an explicit flag to allow overwriting without confirmation.

But I guess people don't seem to care? I've never come across an "mv command considered harmful" essay. Maybe it's time for somebody to write one...

mkdir some_dir mv file.txt some_dir # Put file.txt into the directory

mv other_file.txt new_name.txt # rename other_file.txt to new_name.txt

I love his insights, but I'm not creating an account to see them.

I don't feel like their capabilities are substantially oversold. I think we are shown what they can do, what they can't do, and what they can't do reliably.

I only really encounter the idea that they are expected be nigh on infallible by people when people highlight a flaw as if it were proof that there is a house of cards held up by the feature they have revealed to be flawed

The problems in LLMs are myriad. Finding problems and weaknesses is how they get addressed. They will never be perfect. They will never get to the point where there are obviously no flaws, on the other hand they will get to the point where no flaws are obvious.

Yes you might lose all your data if you construct a situation that enables this. Imagine not having backups of your hard drive. Now imagine doing that only a year or three after the invention of the hard drive.

Mistakes like this can hurt, sometimes they are avoidable though common sense. Sometimes the only way to realise the risk is to be burnt by it.

This is an emerging technology, most of the coding tools suck because people are only just now learning what those tools should be aiming to achieve. Those tools that suck are the data points guiding us to better tools.

Many people expect reat things from AI in the future. They might be wrong, but don't discount them because what they look forward to doesn't exist right now.o

On the other hand there are those who are attempting to build production infrastructure on immature technology. I'm ok with that if their eyes are wide open to the risk they face. Less so if they conceal that risk from their customers.

No one wants monopolies, but the smartest people with infinite resources failing at consumer technology problems is scary when you extrapolate that to existential problem like a meteor.

If you don't like them, simply avoid them and try not to get upset about it. If it's all nonsense it will soon fizzle out. If the potential is realized one can always join in later.

The problem that the author/LLM suggests happened would have resulted in a file or folder called `anuraag_xyz_project` existing in the desktop (being overwritten many times), but the command output shows no such file. I think that's the smoking gun.

Here's one missing piece - when Gemini ran `move * "..\anuraag_xyz project"` it thought (so did the LLM summary) that this would move all files and folders, but in fact this only moves top-level files, no directories. That's probably why after this command it "unexpectedly" found existing folders still there. That's why it then tries to manually move folders.

If the Gemini CLI was actually running the commands it says it was, then there should have been SOMETHING there at the end of all of that moving.

The Gemini CLI repeatedly insists throughout the conversation that "I can only see and interact with files and folders inside the project directory" (despite its apparent willingness to work around its tools and do otherwise), so I think you may be onto something. Not sure how that result in `move`ing files into the void though.

You’re not wrong, but it’s hilarious that the “agentic future” must be wrapped in bubble wrap and safely ensconced in protective cages.

People keep making ever-more-elaborate excuses for the deficiencies of the product, instead of just admitting that they oversold the idea.

[1] https://youtu.be/7gzDC-2ZO8I?feature=shared

Pro tip: you can run `docker diff <container-id>` to see what files have changed in the container since it was created, which can help diagnose unexpected state created by the LLM or anything else.

After that it can continue to refactor the code if some imports need to be modified.

I'll do even more sidetracking and just state that the behaviour of "move" in Windows as described in the article seems absolutely insane.

Edit: so the article links to the documentation for "move" and states that the above is described there. I looked through that page and cannot find any such description - my spider sense is tingling, though I do not now why

I'm just waiting for vibe prompting, where it's arranged for the computer to guess what will make you happy, and then prompt AI agents to do it, no thinking involved at all.

    >move 1 ..\1\
    The system cannot find the path specified.
            0 file(s) moved.