[{"data":1,"prerenderedAt":501},["ShallowReactive",2],{"blog-ru-obedinit-csv-faily-python":3,"blog-counterpart-ru-obedinit-csv-faily-python":490},{"id":4,"title":5,"body":6,"cluster":479,"ctaType":480,"date":481,"description":482,"extension":483,"intent":484,"meta":485,"navigation":103,"ogImage":486,"path":487,"primaryKeyword":488,"readTime":489,"relatedCases":490,"relatedCourse":479,"relatedService":491,"relatedServices":490,"seo":492,"slug":493,"stem":494,"tags":495,"updated":490,"__hash__":500},"blogRu\u002Fru\u002Fblog\u002Fobedinit-csv-faily-python.md","Как объединить CSV-файлы на Python и убрать дубли",{"type":7,"value":8,"toc":474},"minimark",[9,23,26,31,34,44,47,67,70,74,81,388,402,405,409,412,428,431,437,448,451,470],[10,11,12,13,17,18,22],"p",{},"Чтобы ",[14,15,16],"strong",{},"объединить CSV-файлы на Python",", прочитайте их через модуль ",[19,20,21],"code",{},"csv",", проверьте одинаковый набор колонок, нормализуйте ключ записи и сохраните результат в новый файл. Ниже — запускаемый пример: он собирает все CSV из папки, удаляет повторные строки по email и не перезаписывает входные файлы.",[10,24,25],{},"Подход подходит для небольших выгрузок, которые помещаются в память. Для очень больших файлов потребуется потоковая обработка или база данных. Сначала также убедитесь, что у вас есть право обрабатывать эти данные.",[27,28,30],"h2",{"id":29},"подготовьте-папки-и-csv","Подготовьте папки и CSV",[10,32,33],{},"Нужен Python 3.10 или новее. Создайте структуру проекта:",[35,36,42],"pre",{"className":37,"code":39,"language":40,"meta":41},[38],"language-text","csv-report\u002F\n  data\u002F\n    inbox\u002F\n      customers-a.csv\n      customers-b.csv\n    out\u002F\n  merge_csv.py\n","text","",[19,43,39],{"__ignoreMap":41},[10,45,46],{},"У входных файлов должны быть одинаковые заголовки, например:",[35,48,51],{"className":49,"code":50,"language":21,"meta":41,"style":41},"language-csv shiki shiki-themes github-light github-dark","email,name,city\nALICE@example.com,Alice,Minsk\n",[19,52,53,61],{"__ignoreMap":41},[54,55,58],"span",{"class":56,"line":57},"line",1,[54,59,60],{},"email,name,city\n",[54,62,64],{"class":56,"line":63},2,[54,65,66],{},"ALICE@example.com,Alice,Minsk\n",[10,68,69],{},"Разделитель здесь — запятая, кодировка — UTF-8. Если поставщик присылает CSV с другим разделителем или кодировкой, сначала укажите это явно, а не пытайтесь угадать по содержимому.",[27,71,73],{"id":72},"объедините-файлы-и-уберите-повторы","Объедините файлы и уберите повторы",[10,75,76,77,80],{},"Сохраните код в ",[19,78,79],{},"merge_csv.py",":",[35,82,86],{"className":83,"code":84,"language":85,"meta":41,"style":41},"language-python shiki shiki-themes github-light github-dark","import csv\nfrom pathlib import Path\n\n\nROOT = Path(__file__).parent\nINPUT_DIR = ROOT \u002F \"data\" \u002F \"inbox\"\nOUTPUT_FILE = ROOT \u002F \"data\" \u002F \"out\" \u002F \"customers-merged.csv\"\nDEDUPLICATION_KEY = \"email\"\n\n\ndef merge_csv_files() -> int:\n    files = sorted(INPUT_DIR.glob(\"*.csv\"))\n    if not files:\n        raise FileNotFoundError(f\"No CSV files found in {INPUT_DIR}\")\n\n    OUTPUT_FILE.parent.mkdir(parents=True, exist_ok=True)\n    expected_headers: list[str] | None = None\n    seen_keys: set[str] = set()\n    written = 0\n\n    with OUTPUT_FILE.open(\"w\", newline=\"\", encoding=\"utf-8\") as output:\n        writer: csv.DictWriter | None = None\n\n        for file_path in files:\n            with file_path.open(newline=\"\", encoding=\"utf-8-sig\") as source:\n                reader = csv.DictReader(source)\n                headers = reader.fieldnames\n                if not headers or DEDUPLICATION_KEY not in headers:\n                    raise ValueError(f\"{file_path.name}: missing {DEDUPLICATION_KEY!r} column\")\n                if expected_headers is None:\n                    expected_headers = headers\n                    writer = csv.DictWriter(output, fieldnames=headers)\n                    writer.writeheader()\n                elif headers != expected_headers:\n                    raise ValueError(f\"{file_path.name}: CSV columns do not match the first file\")\n\n                for row in reader:\n                    key = (row.get(DEDUPLICATION_KEY) or \"\").strip().casefold()\n                    if not key:\n                        continue\n                    if key in seen_keys:\n                        continue\n                    seen_keys.add(key)\n                    writer.writerow(row)\n                    written += 1\n\n    return written\n\n\nif __name__ == \"__main__\":\n    count = merge_csv_files()\n    print(f\"Saved {count} unique rows to {OUTPUT_FILE}\")\n","python",[19,87,88,93,98,105,110,116,122,128,134,139,144,150,156,162,168,173,179,185,191,197,202,208,214,219,225,231,237,243,249,255,261,267,273,279,285,291,296,302,308,314,320,326,331,337,343,349,354,360,365,370,376,382],{"__ignoreMap":41},[54,89,90],{"class":56,"line":57},[54,91,92],{},"import csv\n",[54,94,95],{"class":56,"line":63},[54,96,97],{},"from pathlib import Path\n",[54,99,101],{"class":56,"line":100},3,[54,102,104],{"emptyLinePlaceholder":103},true,"\n",[54,106,108],{"class":56,"line":107},4,[54,109,104],{"emptyLinePlaceholder":103},[54,111,113],{"class":56,"line":112},5,[54,114,115],{},"ROOT = Path(__file__).parent\n",[54,117,119],{"class":56,"line":118},6,[54,120,121],{},"INPUT_DIR = ROOT \u002F \"data\" \u002F \"inbox\"\n",[54,123,125],{"class":56,"line":124},7,[54,126,127],{},"OUTPUT_FILE = ROOT \u002F \"data\" \u002F \"out\" \u002F \"customers-merged.csv\"\n",[54,129,131],{"class":56,"line":130},8,[54,132,133],{},"DEDUPLICATION_KEY = \"email\"\n",[54,135,137],{"class":56,"line":136},9,[54,138,104],{"emptyLinePlaceholder":103},[54,140,142],{"class":56,"line":141},10,[54,143,104],{"emptyLinePlaceholder":103},[54,145,147],{"class":56,"line":146},11,[54,148,149],{},"def merge_csv_files() -> int:\n",[54,151,153],{"class":56,"line":152},12,[54,154,155],{},"    files = sorted(INPUT_DIR.glob(\"*.csv\"))\n",[54,157,159],{"class":56,"line":158},13,[54,160,161],{},"    if not files:\n",[54,163,165],{"class":56,"line":164},14,[54,166,167],{},"        raise FileNotFoundError(f\"No CSV files found in {INPUT_DIR}\")\n",[54,169,171],{"class":56,"line":170},15,[54,172,104],{"emptyLinePlaceholder":103},[54,174,176],{"class":56,"line":175},16,[54,177,178],{},"    OUTPUT_FILE.parent.mkdir(parents=True, exist_ok=True)\n",[54,180,182],{"class":56,"line":181},17,[54,183,184],{},"    expected_headers: list[str] | None = None\n",[54,186,188],{"class":56,"line":187},18,[54,189,190],{},"    seen_keys: set[str] = set()\n",[54,192,194],{"class":56,"line":193},19,[54,195,196],{},"    written = 0\n",[54,198,200],{"class":56,"line":199},20,[54,201,104],{"emptyLinePlaceholder":103},[54,203,205],{"class":56,"line":204},21,[54,206,207],{},"    with OUTPUT_FILE.open(\"w\", newline=\"\", encoding=\"utf-8\") as output:\n",[54,209,211],{"class":56,"line":210},22,[54,212,213],{},"        writer: csv.DictWriter | None = None\n",[54,215,217],{"class":56,"line":216},23,[54,218,104],{"emptyLinePlaceholder":103},[54,220,222],{"class":56,"line":221},24,[54,223,224],{},"        for file_path in files:\n",[54,226,228],{"class":56,"line":227},25,[54,229,230],{},"            with file_path.open(newline=\"\", encoding=\"utf-8-sig\") as source:\n",[54,232,234],{"class":56,"line":233},26,[54,235,236],{},"                reader = csv.DictReader(source)\n",[54,238,240],{"class":56,"line":239},27,[54,241,242],{},"                headers = reader.fieldnames\n",[54,244,246],{"class":56,"line":245},28,[54,247,248],{},"                if not headers or DEDUPLICATION_KEY not in headers:\n",[54,250,252],{"class":56,"line":251},29,[54,253,254],{},"                    raise ValueError(f\"{file_path.name}: missing {DEDUPLICATION_KEY!r} column\")\n",[54,256,258],{"class":56,"line":257},30,[54,259,260],{},"                if expected_headers is None:\n",[54,262,264],{"class":56,"line":263},31,[54,265,266],{},"                    expected_headers = headers\n",[54,268,270],{"class":56,"line":269},32,[54,271,272],{},"                    writer = csv.DictWriter(output, fieldnames=headers)\n",[54,274,276],{"class":56,"line":275},33,[54,277,278],{},"                    writer.writeheader()\n",[54,280,282],{"class":56,"line":281},34,[54,283,284],{},"                elif headers != expected_headers:\n",[54,286,288],{"class":56,"line":287},35,[54,289,290],{},"                    raise ValueError(f\"{file_path.name}: CSV columns do not match the first file\")\n",[54,292,294],{"class":56,"line":293},36,[54,295,104],{"emptyLinePlaceholder":103},[54,297,299],{"class":56,"line":298},37,[54,300,301],{},"                for row in reader:\n",[54,303,305],{"class":56,"line":304},38,[54,306,307],{},"                    key = (row.get(DEDUPLICATION_KEY) or \"\").strip().casefold()\n",[54,309,311],{"class":56,"line":310},39,[54,312,313],{},"                    if not key:\n",[54,315,317],{"class":56,"line":316},40,[54,318,319],{},"                        continue\n",[54,321,323],{"class":56,"line":322},41,[54,324,325],{},"                    if key in seen_keys:\n",[54,327,329],{"class":56,"line":328},42,[54,330,319],{},[54,332,334],{"class":56,"line":333},43,[54,335,336],{},"                    seen_keys.add(key)\n",[54,338,340],{"class":56,"line":339},44,[54,341,342],{},"                    writer.writerow(row)\n",[54,344,346],{"class":56,"line":345},45,[54,347,348],{},"                    written += 1\n",[54,350,352],{"class":56,"line":351},46,[54,353,104],{"emptyLinePlaceholder":103},[54,355,357],{"class":56,"line":356},47,[54,358,359],{},"    return written\n",[54,361,363],{"class":56,"line":362},48,[54,364,104],{"emptyLinePlaceholder":103},[54,366,368],{"class":56,"line":367},49,[54,369,104],{"emptyLinePlaceholder":103},[54,371,373],{"class":56,"line":372},50,[54,374,375],{},"if __name__ == \"__main__\":\n",[54,377,379],{"class":56,"line":378},51,[54,380,381],{},"    count = merge_csv_files()\n",[54,383,385],{"class":56,"line":384},52,[54,386,387],{},"    print(f\"Saved {count} unique rows to {OUTPUT_FILE}\")\n",[10,389,390,393,394,397,398,401],{},[19,391,392],{},"utf-8-sig"," читает обычный UTF-8 и убирает BOM, который иногда добавляют табличные редакторы. ",[19,395,396],{},"casefold()"," сравнивает email без различия регистра, а ",[19,399,400],{},"strip()"," убирает пробелы по краям. Пустые ключи пропускаются, потому что без ключа надёжно определить повтор нельзя.",[10,403,404],{},"Скрипт сохраняет первую встреченную строку для каждого email. Если нужно оставить самую свежую запись, добавьте явное правило сравнения даты и протестируйте его на примерах. Не выбирайте «последнюю» строку случайно: порядок файлов сам по себе не доказывает, какая запись новее.",[27,406,408],{"id":407},"запустите-и-проверьте-результат","Запустите и проверьте результат",[10,410,411],{},"Из папки проекта выполните:",[35,413,417],{"className":414,"code":415,"language":416,"meta":41,"style":41},"language-bash shiki shiki-themes github-light github-dark","python merge_csv.py\n","bash",[19,418,419],{"__ignoreMap":41},[54,420,421,424],{"class":56,"line":57},[54,422,85],{"class":423},"sScJk",[54,425,427],{"class":426},"sZZnC"," merge_csv.py\n",[10,429,430],{},"Ожидаемый вывод при двух уникальных клиентах:",[35,432,435],{"className":433,"code":434,"language":40,"meta":41},[38],"Saved 2 unique rows to ...\u002Fdata\u002Fout\u002Fcustomers-merged.csv\n",[19,436,434],{"__ignoreMap":41},[10,438,439,440,443,444,447],{},"Итоговый файл появится в ",[19,441,442],{},"data\u002Fout\u002F",". Исходные CSV останутся в ",[19,445,446],{},"data\u002Finbox\u002F",". Повторный запуск создаёт новый отчёт заново, поэтому он не добавляет те же строки в конец результата.",[10,449,450],{},"Перед регулярным запуском проверьте три случая: повторяющийся email в двух файлах, пустой email и файл с отличающимися заголовками. Для настоящих рабочих данных полезно добавить журнал запуска, число пропущенных строк и уведомление при ошибке, не записывая в лог персональные значения.",[10,452,453,454,459,460,464,465,469],{},"Если исходные CSV большие, набор колонок меняется или результат нужно регулярно отправлять в CRM, вынесите правила нормализации в отдельные функции и добавьте тесты на реальные крайние случаи. В ",[455,456,458],"a",{"href":457},"\u002Fru\u002Fcourses\u002Fpython-automation\u002F","курсе по автоматизации Python"," такой CSV-процесс входит в практическую траекторию; более сложный сбор данных с сайтов разобран в статье про ",[455,461,463],{"href":462},"\u002Fru\u002Fblog\u002Fparsers-scrapy-python\u002F","парсеры на Python",", а настройка ежедневного выполнения — в руководстве по ",[455,466,468],{"href":467},"\u002Fru\u002Fblog\u002Fzapusk-python-po-raspisaniyu-linux\u002F","запуску Python по расписанию через systemd",".",[471,472,473],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}",{"title":41,"searchDepth":63,"depth":63,"links":475},[476,477,478],{"id":29,"depth":63,"text":30},{"id":72,"depth":63,"text":73},{"id":407,"depth":63,"text":408},"python-automation","course","2026-10-12","Пошаговый пример на Python: собрать CSV-файлы с одинаковыми колонками, нормализовать email, удалить дубли и сохранить отдельный отчёт без изменения исходников.","md","educational",{},"\u002Fimages\u002Fog\u002Fblog\u002Fru\u002Fobedinit-csv-faily-python.png","\u002Fru\u002Fblog\u002Fobedinit-csv-faily-python","объединить CSV файлы Python","6 мин",null,"automation",{"title":5,"description":482},"obedinit-csv-faily-python","ru\u002Fblog\u002Fobedinit-csv-faily-python",[496,497,498,499],"Python","CSV","Автоматизация","Данные","U8ELURQ3ZrERt4GnfdWbZANTvw0GRgssxSEkkTNvtks",1791825742600]