{"success":true,"count":2,"items":[{"videoId":"8S0-C75Yhuc","chunkIndex":0,"totalChunks":2,"title":"프롬프트에 ‘잘하라’고 써도 모델은 못 한다 (Anthropic) — Part 1 of 2","thumbnail":"https://i.ytimg.com/vi/8S0-C75Yhuc/maxresdefault.jpg","duration":954,"uploader":"Recent AI","youtubeUrl":"https://www.youtube.com/watch?v=8S0-C75Yhuc","keywords":["prompt-engineering","llm-evaluation","ai-safety","instruction-following","tool-use","version-control","model-migration","customer-support"],"normalizedKeywords":["엔지니어링","기술 트렌드","교육"],"targetAudience":[{"who":"프롬프트 엔지니어","why":"프롬프트 구조화, 평가, 디버깅의 실전 원칙을 배울 수 있음"},{"who":"AI 제품 담당자","why":"모델 교체 시 성능 저하 원인과 대응법을 이해할 수 있음"},{"who":"개발자","why":"도구 사용과 캡 능력 한계를 고려한 구현 힌트를 얻을 수 있음"}],"normalizedAudience":["엔지니어·개발자","프로덕트 매니저·기획자","학생·주니어"],"summary":"이 영상은 복잡하게 누적된 프롬프트를 새 모델로 옮길 때 왜 성능이 흔들리는지, 그리고 무엇을 기준으로 고쳐야 하는지를 설명한다. 핵심은 '잘하라'는 식의 모호한 지시보다, 구조를 분리하고 평가를 붙여 어떤 변경이 실제 개선인지 검증해야 한다는 점이다. 역할, 정책, 지침, 데이터가 뒤섞인 프롬프트를 정리하면 모델이 이해하기 쉬워지고, 실제 평가 점수도 개선될 수 있음을 보여준다.\n\n또한 모델이 단순히 환각을 줄여야 하는 게 아니라, 이미 알고 있는 정보를 불필요한 방어 규칙 때문에 숨기는 문제도 있다는 점을 짚는다. 과거 모델의 버그를 막기 위해 넣은 패치가 새 모델에서는 오히려 과잉 최적화가 될 수 있으므로, 버전 관리와 회귀 테스트가 중요하다고 강조한다. 마지막으로 계산처럼 능력이 필요한 작업은 지시만으로 해결되지 않으며, 도구를 붙여 모델이 실제로 수행할 수 있게 만들어야 한다는 실용적 원칙을 제시한다.","insights":["프롬프트 성능은 구조가 무너지면 함께 무너진다.","평가 없이는 개선인지 우연인지 구분할 수 없다.","과거 모델용 방어문구가 새 모델에선 독이 될 수 있다.","지시만으로 안 되면 능력을 주는 도구가 필요하다."],"keyClips":[{"clipId":"8S0-C75Yhuc:c0:1-13","startSegmentIndex":1,"endSegmentIndex":13,"startTime":2.59,"endTime":101.2,"durationSeconds":98.6,"preview":"프롬프트는 구조가 전부","mustSee":true},{"clipId":"8S0-C75Yhuc:c0:14-30","startSegmentIndex":14,"endSegmentIndex":30,"startTime":101.2,"endTime":265.68,"durationSeconds":164.5,"preview":"정리만 해도 좋아진다","mustSee":false},{"clipId":"8S0-C75Yhuc:c0:31-48","startSegmentIndex":31,"endSegmentIndex":48,"startTime":265.68,"endTime":441.83,"durationSeconds":176.1,"preview":"오답은 과잉방어다","mustSee":false},{"clipId":"8S0-C75Yhuc:c0:49-64","startSegmentIndex":49,"endSegmentIndex":64,"startTime":441.83,"endTime":603.32,"durationSeconds":161.5,"preview":"능력은 지시로 안 생긴다","mustSee":true}],"curatedSegments":[{"segmentIndex":7,"text":"We need evaluations to provide that rigor um to understand whether a change to our prompt is actually correlating to an improvement in its performance.","startTime":44.28,"endTime":56.24,"durationSeconds":12,"level":"B2","overallScore":6.4,"rationale":""},{"segmentIndex":13,"text":"So, we need to have an eval suite to act as a way of testing that regression, so that we can apply our prompting best practices to that.","startTime":88.88,"endTime":101.2,"durationSeconds":12,"level":"B2","overallScore":6.6,"rationale":""},{"segmentIndex":30,"text":"A general rule of thumb that I like to follow is if you're reading a prompt and you can't tell guidelines from policy from data, most likely the model isn't able to either.","startTime":253.4,"endTime":265.68,"durationSeconds":12,"level":"B2","overallScore":6.6,"rationale":""},{"segmentIndex":45,"text":"So, it's likely that instructions like these have now become redundant and are actually being overfitted to. So, what we're going to tell the model instead is give this balanced view uh um where it says, you know,\"Customers on grandfather's plan have different allowances, but it's captured in the customer information that's given, and that is the accurate source of truth.\"","startTime":392.12,"endTime":402.83,"durationSeconds":11,"level":"C1","overallScore":6.4,"rationale":""},{"segmentIndex":60,"text":"Now, telling the model to do a good job isn't particularly helpful when we don't give the model the capability to actually do a good job.","startTime":560.8389999999999,"endTime":570.92,"durationSeconds":10,"level":"B2","overallScore":6.2,"rationale":""},{"segmentIndex":4,"text":"Um it's built up and it's complex. And when we're migrating to a new model, we're finding that suddenly a lot of our test cases are no longer working as well as we expected.","startTime":21.48,"endTime":34,"durationSeconds":13,"level":"B2","overallScore":5.6,"rationale":""},{"segmentIndex":9,"text":"And when you migrate to a different model, it could be that your system is no longer working as well for two reasons.","startTime":62.56,"endTime":70.2,"durationSeconds":8,"level":"B2","overallScore":5.6,"rationale":""},{"segmentIndex":12,"text":"The second case is where actually the model that we're changing to isn't as capable, and no amount of prompting is going to fix that.","startTime":80.8,"endTime":88.88,"durationSeconds":8,"level":"B2","overallScore":5.8,"rationale":""},{"segmentIndex":20,"text":"When we look at the instructions here, they're all grouped into one big paragraph. So, we've got some reasoning here. We've got instructions about the role um some critical instructions as well without a real way of unpacking um policy from guidelines from tone, etc.","startTime":135.43,"endTime":162.07999999999998,"durationSeconds":27,"level":"B2","overallScore":5.6,"rationale":""},{"segmentIndex":29,"text":"And this is a best practice that you can return to at any stage of writing and maintaining your prompt, especially as your prompts get more detailed and more complex.","startTime":243.28,"endTime":253.4,"durationSeconds":10,"level":"B2","overallScore":5.6,"rationale":""},{"segmentIndex":49,"text":"Well, we worry a lot about hallucinations or the invention of facts and numbers, but actually the opposite can also happen.","startTime":441.83,"endTime":452.36,"durationSeconds":11,"level":"B2","overallScore":5.4,"rationale":""},{"segmentIndex":1,"text":"Imagine that we have a prompt that multiple people have been collaborating on, contributing to. There's no clear owner.","startTime":2.59,"endTime":8.6,"durationSeconds":6,"level":"B1","overallScore":4.6,"rationale":""},{"segmentIndex":11,"text":"And therefore, we can tune our prompting to fix that behavior.","startTime":76.56,"endTime":80.8,"durationSeconds":4,"level":"B1","overallScore":4.8,"rationale":""},{"segmentIndex":22,"text":"So, what we've done is first of all added some structure. So, you can see that we've added XML tags here to define the role, to separate general guidelines, to separate policy, to separate tone of voice, um etc. So, I preempted some changes we want to make to this prompt.","startTime":165.28,"endTime":189.59,"durationSeconds":24,"level":"B2","overallScore":5,"rationale":""},{"segmentIndex":24,"text":"So, we can see just by clearing up the prompt, we've already improved the model's performance on this prepaid scenario.","startTime":206.48,"endTime":214.68,"durationSeconds":8,"level":"B1","overallScore":4.4,"rationale":""},{"segmentIndex":26,"text":"There's going to be some natural level of variance in the different runs of the eval, and we'll come back to that case specifically to see if we can make the prompt consistently better in that area. So, what did we learn from this then?","startTime":220.76,"endTime":233,"durationSeconds":12,"level":"B2","overallScore":4.6,"rationale":""},{"segmentIndex":33,"text":"And the reason this is a slightly complex case is because the customer test case that we're dealing with is on a legacy plan.","startTime":280,"endTime":287.08,"durationSeconds":7,"level":"B2","overallScore":4.6,"rationale":""},{"segmentIndex":42,"text":"So, it's clear that this instruction, this latter one,\"Never give customer the wrong information,\"is the instruction that the bot has been optimizing for.","startTime":362.44,"endTime":373,"durationSeconds":11,"level":"B2","overallScore":4.4,"rationale":""},{"segmentIndex":50,"text":"The model can withhold information that it actually has access to.","startTime":452.36,"endTime":457.44,"durationSeconds":5,"level":"B1","overallScore":4.4,"rationale":""},{"segmentIndex":51,"text":"Now, we saw here that this is likely a result of a patch that we introduced for a previous model and a best practice that we could follow here is actually using version control.","startTime":457.44,"endTime":468.04,"durationSeconds":11,"level":"B2","overallScore":4.6,"rationale":""}],"generatedAt":"2026-07-06T02:41:58.897Z","keyClipsTotalSec":545},{"videoId":"8S0-C75Yhuc","chunkIndex":1,"totalChunks":2,"title":"프롬프트에 ‘잘하라’고 써도 모델은 못 한다 (Anthropic) — Part 2 of 2","thumbnail":"https://i.ytimg.com/vi/8S0-C75Yhuc/maxresdefault.jpg","duration":954,"uploader":"Recent AI","youtubeUrl":"https://www.youtube.com/watch?v=8S0-C75Yhuc","keywords":["prompt-engineering","llm-evaluation","ai-agents","tool-use","instruction-following","automation","optimization"],"normalizedKeywords":["엔지니어링","프로덕트","기술 트렌드"],"targetAudience":[{"who":"프롬프트 엔지니어","why":"지시문만으로 해결되지 않는 한계를 줄이는 설계법을 배울 수 있음"},{"who":"AI 제품 담당자","why":"평가-수정 루프와 툴 사용으로 품질을 안정화하는 방법이 유용함"},{"who":"LLM 앱 개발자","why":"모델에 맡기지 말고 도구와 분리된 단계로 처리하는 구조를 얻을 수 있음"},{"who":"실무 자동화 담당자","why":"런타임 제약과 소프트 규칙을 유연하게 반영하는 운영 방식을 참고할 수 있음"}],"normalizedAudience":["엔지니어·개발자","프로덕트 매니저·기획자","지식노동자 일반"],"summary":"이 영상은 LLM에게 '잘하라'고 지시하는 것만으로는 능력이 늘지 않으며, 필요한 것은 툴 제공과 문제를 쪼개는 구조라는 점을 반복해서 보여준다. 계산 문제에서는 모델이 정신연산을 잘하게 만드는 대신 도구를 붙여 정답을 안정적으로 내게 하고, 고객 응대 문제에서는 한쪽 면만 강조한 지시가 오히려 잘못된 행동을 유도한다는 점을 짚는다.\n\n후반부에서는 하나의 큰 프롬프트보다 생성-평가-수정(generate-evaluate-repair) 루프를 분리해 운영하는 접근이 더 적은 토큰과 낮은 지연으로 더 좋은 결과를 낸다고 설명한다. 또한 이런 구조를 쓰면 런타임에서 소프트 제약을 주입해 케이스별 요구를 유연하게 반영할 수 있어, 실제 업무 자동화에 더 적합하다는 메시지를 전달한다.","insights":["지시는 능력을 만들지 않고, 도구가 능력을 만든다.","한쪽만 강조한 규칙은 모델을 쉽게 오버핏시킨다.","복잡한 작업은 한 번에보다 분리된 루프가 더 안정적이다.","생성·평가·수정 분리는 품질과 효율을 함께 올린다.","소프트 제약은 코드 수정 없이 런타임에서 다루는 게 좋다."],"keyClips":[{"clipId":"8S0-C75Yhuc:c1:1-5","startSegmentIndex":1,"endSegmentIndex":5,"startTime":600.83,"endTime":660.36,"durationSeconds":59.5,"preview":"지시보다 도구","mustSee":false},{"clipId":"8S0-C75Yhuc:c1:6-17","startSegmentIndex":6,"endSegmentIndex":17,"startTime":660.36,"endTime":815.79,"durationSeconds":155.4,"preview":"오버핏된 규칙 수정","mustSee":true},{"clipId":"8S0-C75Yhuc:c1:18-25","startSegmentIndex":18,"endSegmentIndex":25,"startTime":815.79,"endTime":892.52,"durationSeconds":76.7,"preview":"생성평가수정 루프","mustSee":true},{"clipId":"8S0-C75Yhuc:c1:26-34","startSegmentIndex":26,"endSegmentIndex":34,"startTime":892.52,"endTime":945.28,"durationSeconds":52.8,"preview":"런타임 소프트제약","mustSee":true}],"curatedSegments":[{"segmentIndex":7,"text":"We need evaluations to provide that rigor um to understand whether a change to our prompt is actually correlating to an improvement in its performance.","startTime":44.28,"endTime":56.24,"durationSeconds":12,"level":"B2","overallScore":6.4,"rationale":""},{"segmentIndex":13,"text":"So, we need to have an eval suite to act as a way of testing that regression, so that we can apply our prompting best practices to that.","startTime":88.88,"endTime":101.2,"durationSeconds":12,"level":"B2","overallScore":6.6,"rationale":""},{"segmentIndex":30,"text":"A general rule of thumb that I like to follow is if you're reading a prompt and you can't tell guidelines from policy from data, most likely the model isn't able to either.","startTime":253.4,"endTime":265.68,"durationSeconds":12,"level":"B2","overallScore":6.6,"rationale":""},{"segmentIndex":45,"text":"So, it's likely that instructions like these have now become redundant and are actually being overfitted to. So, what we're going to tell the model instead is give this balanced view uh um where it says, you know,\"Customers on grandfather's plan have different allowances, but it's captured in the customer information that's given, and that is the accurate source of truth.\"","startTime":392.12,"endTime":402.83,"durationSeconds":11,"level":"C1","overallScore":6.4,"rationale":""},{"segmentIndex":60,"text":"Now, telling the model to do a good job isn't particularly helpful when we don't give the model the capability to actually do a good job.","startTime":560.8389999999999,"endTime":570.92,"durationSeconds":10,"level":"B2","overallScore":6.2,"rationale":""},{"segmentIndex":4,"text":"Um it's built up and it's complex. And when we're migrating to a new model, we're finding that suddenly a lot of our test cases are no longer working as well as we expected.","startTime":21.48,"endTime":34,"durationSeconds":13,"level":"B2","overallScore":5.6,"rationale":""},{"segmentIndex":9,"text":"And when you migrate to a different model, it could be that your system is no longer working as well for two reasons.","startTime":62.56,"endTime":70.2,"durationSeconds":8,"level":"B2","overallScore":5.6,"rationale":""},{"segmentIndex":12,"text":"The second case is where actually the model that we're changing to isn't as capable, and no amount of prompting is going to fix that.","startTime":80.8,"endTime":88.88,"durationSeconds":8,"level":"B2","overallScore":5.8,"rationale":""},{"segmentIndex":20,"text":"When we look at the instructions here, they're all grouped into one big paragraph. So, we've got some reasoning here. We've got instructions about the role um some critical instructions as well without a real way of unpacking um policy from guidelines from tone, etc.","startTime":135.43,"endTime":162.07999999999998,"durationSeconds":27,"level":"B2","overallScore":5.6,"rationale":""},{"segmentIndex":29,"text":"And this is a best practice that you can return to at any stage of writing and maintaining your prompt, especially as your prompts get more detailed and more complex.","startTime":243.28,"endTime":253.4,"durationSeconds":10,"level":"B2","overallScore":5.6,"rationale":""},{"segmentIndex":49,"text":"Well, we worry a lot about hallucinations or the invention of facts and numbers, but actually the opposite can also happen.","startTime":441.83,"endTime":452.36,"durationSeconds":11,"level":"B2","overallScore":5.4,"rationale":""},{"segmentIndex":1,"text":"Imagine that we have a prompt that multiple people have been collaborating on, contributing to. There's no clear owner.","startTime":2.59,"endTime":8.6,"durationSeconds":6,"level":"B1","overallScore":4.6,"rationale":""},{"segmentIndex":11,"text":"And therefore, we can tune our prompting to fix that behavior.","startTime":76.56,"endTime":80.8,"durationSeconds":4,"level":"B1","overallScore":4.8,"rationale":""},{"segmentIndex":22,"text":"So, what we've done is first of all added some structure. So, you can see that we've added XML tags here to define the role, to separate general guidelines, to separate policy, to separate tone of voice, um etc. So, I preempted some changes we want to make to this prompt.","startTime":165.28,"endTime":189.59,"durationSeconds":24,"level":"B2","overallScore":5,"rationale":""},{"segmentIndex":24,"text":"So, we can see just by clearing up the prompt, we've already improved the model's performance on this prepaid scenario.","startTime":206.48,"endTime":214.68,"durationSeconds":8,"level":"B1","overallScore":4.4,"rationale":""},{"segmentIndex":26,"text":"There's going to be some natural level of variance in the different runs of the eval, and we'll come back to that case specifically to see if we can make the prompt consistently better in that area. So, what did we learn from this then?","startTime":220.76,"endTime":233,"durationSeconds":12,"level":"B2","overallScore":4.6,"rationale":""},{"segmentIndex":33,"text":"And the reason this is a slightly complex case is because the customer test case that we're dealing with is on a legacy plan.","startTime":280,"endTime":287.08,"durationSeconds":7,"level":"B2","overallScore":4.6,"rationale":""},{"segmentIndex":42,"text":"So, it's clear that this instruction, this latter one,\"Never give customer the wrong information,\"is the instruction that the bot has been optimizing for.","startTime":362.44,"endTime":373,"durationSeconds":11,"level":"B2","overallScore":4.4,"rationale":""},{"segmentIndex":50,"text":"The model can withhold information that it actually has access to.","startTime":452.36,"endTime":457.44,"durationSeconds":5,"level":"B1","overallScore":4.4,"rationale":""},{"segmentIndex":51,"text":"Now, we saw here that this is likely a result of a patch that we introduced for a previous model and a best practice that we could follow here is actually using version control.","startTime":457.44,"endTime":468.04,"durationSeconds":11,"level":"B2","overallScore":4.6,"rationale":""}],"generatedAt":"2026-07-06T21:32:30.651Z","keyClipsTotalSec":545}]}