Jane Street: Socratic Tutor

I've been experimenting with a lot of these learning skills (see also, e.g., https://github.com/mattpocock/skills/blob/main/skills/produc..., which I think addresses the progress tracking part better than this skill) and they're great for getting simple answers, getting unstuck on exercises, getting some intuition through simulations, etc, but I still feel some more structure is needed for effective self-teaching. Has anyone tried using a hard book as a sort of course structure and asking the LLM to teach you through it? You could of course just read the book, but a lot of advanced subjects don't have great learning materials.

This is really cool. I think we will see more of these, inching towards 'refined' thought and expression percolating from AI models into human minds.

I noticed that the quality of email text has gone up around my work these days. everyone is sounding like a master of drafting emails. Ofcourse Ai has a role in it. But real cool thing about this is, this is pulling up people's thinking to have a structure, minimalism and sequencing of expression. Same as how textile industry has enabled people to have an style and expression in clothing.

Socratic questions are not asked blindly. They are meant to help the student to realize the incompleteness or falsety of what they think they know. That us, to realise they do not actually know something and thus put them into a position of not knowing, which is the only way to gain understanding.

One man said, that enlightement (as in Zen) cannot be faked. Whoever had it, will be able to tell if someone else had it or not. There is no way to merely sound like it. Socratic method is like enlightement.

I’ve struggled to get AI to teach me things well without spilling hints or giving answers away. Assuming this was built on some solid evals this could be awesome.

> When the user is wrong, do not give the answer. Ask a narrower question that exposes the contradiction in their reasoning

Socratic tutoring is the first thing I did when ChatGPT first shipped. Your liberal arts education is a "huge unlock" if you just apply it.

See also https://en.wikipedia.org/wiki/Mastery_learning

https://en.wikipedia.org/wiki/Programmed_learning

Make your own custom skinner boxes. https://en.wikipedia.org/wiki/Operant_conditioning_chamber

Oh this is so good. I asked it to teach me programming in Logo; though the model is not the best (GLM 4.7, on pi), the tutor is asking very good questions and is not too pedantic about exact answers.

For those new (beginners), install pi from pi.dev and then your favorite AI model (ask your model how to set up pi and it). Then ask your pi-based AI how to set up the tutor (just give it this link, it will set it up for you).

Then start pi by typing pi in the terminal; and to launch the tutor preface your question with: /skill:socratic-tutor. Like so:

/skill:socratic-tutor teach me Python.

This is a cool idea. Though trying this out using the claude CLI is kind of funny because by default it's suggestion for the next prompt is the answer to the question.

Socratic tutoring with LLMs works great for me and I have used this skill and it also works great (can you do the same with a simple prompt? probably)

Using LLMs this way was really refreshing, I could really feel how I started to think on my own again.

Been using this with Opus 5.5 on Medium for a week or so. I've used the same model for learning before and I definitely prefer using this skill because it stops the model from the usual handholding. Although I'd say most of the effect can be replicated with this line: "When the user is wrong, do not give the answer. Ask a narrower question that exposes the contradiction in their reasoning."

I love the idea but Jane Street might have copied (or independently rediscovered) https://github.com/bevibing/socrates-skill

I want an AI that I can give a paper, and then sits with me to understand it, maybe even implement it (though this is the easy part for the LLM).

what am i missing, why does it have 434 forks and 6.3k starts? is it botted? what's the point

I'd be curious to see whether learners can still solve similar problems on their own a week later

Love this. There is so much value when using LLMs this way, it needs effort and consistency from the user but the results are really good.

A vanilla version of this is, at the end of a session, ask the model to generate a couple of "ultimate" questions to really flex your understanding. Obviously, the quality depends on the model but I've found it useful.

Reminds me of Keating AI. Highly recommend: https://keating.help/

Has anyone tried ChatGPT's study mode? I thought it was supposed to be just like this, but I was underwhelmed when I tried it, and they ended up removing it. (It might be back now?)

I have the leaked prompt around here somewhere...

I’ve been building a harness for just this all year, they’re great. One of the main use cases for LLMs I think is largely overlooked.

Socratic tutor skill isn't new at all, why did this reach the front page

man people need to talk to people this is depressing

Trying it right now. it's pretty decent and it does sort of make my brain work. Pondering right now when the "bitter lesson" will make these skills useless.

I've tried something like this and ran into a problem which I'm not seeing a fix for. Generally, to really apply this method, you need to understand the person asking the question.

>Tell the user that you will start by asking a few questions, one at a time, to see what they already know.

This puts it into the session each time. Do this once and it seems to work. By the 5th time, you are getting tired of reconfirming your mastery. This ends up depending upon the user to have a pretty solid foundation of what they know and then use this skill to expand the boundary a bit, and even then the AI can end up making so many assumptions that a person has trouble following it along.

I was thinking some sort of concept mapping kept long term might help, but such a mapping can become really difficult to build. Which should be expected when we consider that teaching isn't just its own discipline, but has many sub-disciplines targeted to specific areas (even if some of it, like classroom management, doesn't apply to this scenario).

I made a few attempts at a knowledge tracking tool for personal use to track how much I knew Japanese beyond a simple JLPT level or similar with the plan to tie it into something like this, but it wasn't great. I could 'cheat' and just tap into my Wanikani API to see my progress, but that still skips so many other areas of Japanese, and even when WaniKani says something is burned, I really want AI to still use it until I truly internalize it like I do words in English. I suspect other areas of knowledge are similar.

This works really well.

A positive take: this works much better than the many projects on gh that did this before Jane Street. This would be evidence that creativity and execution still matter, even when the product is just an md file

A cynical take: network effects are everything. tens of projects doing the same thing and this only gains traction when an already popular org does it

someone at jane street likes https://www.mathacademy.com/how-it-works

How is this even remotely new?

I remember with ChatGPT 3.5 people were saying "LLM socratic tutor will revolutionize education"

Here we are 3 years later (almost to the day) and well, education isn't revolutionized in the way people imagined.

I see a lot of this at my company too - everyone re-inventing the same ideas over and over. I wonder how much people actually adopt other teams' implementation of this. In my experience I'm bombarded with a thousand different iterations of these things that everyone wants to advertise as a sort of "Look at me! I'm so AI!" but I simply don't have time to evaluate every different persons variation on these things, I certainly don't have time to really thoroughly benchmark them, and from experience a lot what I needed a carefully tuned skill to achieve 6 months ago is trivial for a better tuned agent today. I'm literally getting to the point where even my own skills are beginning to bit-rot.

If it was this easy, educators would have written up sets of standard Socratic teaching prompts long ago. Like perhaps in the time of Socrates himself. Or the British empire or John Dewey etc. There is probably some great research in the field of Education on this approach.

LLMs should first learn how to write English for humans.

Right now what they produce reads like legalese, or patents.

I suspect this is because the AI labs are optimizing for agents talking to agents, rather than AI talking to humans (which is harder to optimize, and does not improve intelligence as fast).

Sensei, please teach me the ways to reach alpha just like you.

Jokes aside, pretty cool. This is just a skill, and it'd be easy to dismiss it. But I do want to become a better horse whisperer. Prompt engineering used to be mocked a lot, especially on HN, and skills really is just that but with a new branding.

I think Duolingo's days and the likes are counted, AI is just a great way to quickly get up to speed on a subject or a language.

I assume this isn't going to cause the LLM to start jumping me with questions about the nature of piety or justice

This comes off as AI consultant slop from a year ago. I wonder who is in charge of AI at Jane Street.

My client just got done firing someone who thought filling the codebase up with files like this would be a good idea.

It would seem the AI revolution is happening at wildly different rates depending on where you are standing.

I think it's a great idea, but I'm somewhat anti-skill.

Every single skill I see just strikes me as a prompt. In every case it's easier for me to create a description of what is needed specific to the task. If I need a longstanding process I will discuss with the agent and create a document.

This is one of the most flexible tools ever made, and I frankly am not going to accept unnecessary systematization of how I use it.

Ah yes, I recall all those times as an engineer where my managers kept insisting I communicate like Socrates to improve my work. Software is solved.

I WANT TO SEE THIS COMMENT ON COMMENTS

This is honestly pretty damn good actually.

Running it with my local models it is a pretty helpful way to study up something where you already have some bases.

I dunno if you really need this. I asked AI to generate me a Lean tutorial with exercises and it did a really great job, one-shot with no skills. My 4th attempt to learn Lean and I'm actually making progress this time. (Mostly because you can say "I'm a bit stuck here, can you give me a hint" - it's really good at Lean even without access to the Lean compiler.)

Maybe it improves things a bit though. I might try it next time.

can i use this for learning DSA

At some point the context will degrade and the questions might not be useful.

One more interesting test I'd like to do with this skill is. "Is justified true belief knowledge?" and arrive at Gettier through Socratic questioning.

brainrot

[flagged]

[dead]

[dead]

[dead]

[flagged]