{"id":476,"date":"2023-02-16T10:56:15","date_gmt":"2023-02-16T10:56:15","guid":{"rendered":"https:\/\/tinyytopic.com\/?p=476"},"modified":"2023-02-16T10:56:15","modified_gmt":"2023-02-16T10:56:15","slug":"remove-empty-and-duplicates-lines-in-a-file","status":"publish","type":"post","link":"https:\/\/tinyytopic.com\/index.php\/2023\/02\/16\/remove-empty-and-duplicates-lines-in-a-file\/","title":{"rendered":"Remove empty and duplicates lines in a file"},"content":{"rendered":"\n

How to remove empty and duplicate lines in a file?<\/mark><\/h5><\/div>\n\n\n\n

The objective of this Python function is to read the contents of a text file, remove any empty lines and any duplicate lines, and then overwrite the file with the updated content that no longer contains empty lines or duplicate lines.<\/p>\n\n\n\n

Ready-to-use Python function to empty or blank lines and duplicate lines in a file:<\/p>\n\n\n\n

def remove_empty_duplicate_lines_text_file(txtFile):\n    # Remove empty lines and duplicate lines from the file\n    output = ''\n    lines_seen = []\n    \n    with open(txtFile, 'r', encoding=\"utf-8\") as file:\n        for line in file:\n            if not line.isspace() and line.replace(\"\\n\", \"\") not in lines_seen:\n                output+=line\n                lines_seen.append(line.replace(\"\\n\", \"\"))\n    if output[-1] == '\\n': output = output[:-1] # remove newline character if exists at end of variable\n\n    file = open(txtFile, 'w+', encoding=\"utf-8\")\n    file.write(output)\n    file.close()<\/pre>\n\n\n\n
Write your main code as a sample below,<\/p>\n\n\n\n
remove_empty_duplicate_lines_text_file(\"texts to remove blank and duplicate lines.txt\")<\/code><\/pre>\n\n\n\n
The output of the code is (check the text file after executing the code),<\/p>\n\n\n\n

Text file content before executing the function:<\/mark><\/p>\n\n\n\n
import requests\n\n# Define headers\n# Define the API endpoint\nurl = \"https:\/\/api.kite.trade\/instruments\" \n# Define headers\n\n# Define headers\n<\/code><\/pre>\n\n\n\n
Text file content after executing the function:<\/mark><\/p>\n\n\n\n
import requests\n# Define headers\n# Define the API endpoint\nurl = \"https:\/\/api.kite.trade\/instruments\" <\/code><\/pre>\n\n\n\n
How does the function work?<\/mark><\/h5><\/div>\n\n\n\n
The Python function remove_empty_duplicate_lines_text_file<\/code> takes one argument, txtFile<\/code>, which is a string representing the path to a text file. The function’s goal is to read the contents of the file, remove any empty lines and any duplicate lines, and then overwrite the file with the updated content that no longer contains empty lines or duplicate lines.<\/p>\n\n\n\n
Here is a breakdown of how the function works:<\/p>\n\n\n\n
    \n
  1. The function initializes an empty string called output<\/code> to hold the updated content of the text file.<\/li>\n\n\n\n
  2. The function also initializes an empty list called lines_seen<\/code> to keep track of the lines that have already been seen.<\/li>\n\n\n\n
  3. The with<\/code> statement is used to open the text file specified in txtFile<\/code> and create a file object called file<\/code>.<\/li>\n\n\n\n
  4. The for<\/code> loop iterates over each line in the file.<\/li>\n\n\n\n
  5. The first if<\/code> statement checks if the line is not a whitespace character, using the isspace()<\/code> method of the string.<\/li>\n\n\n\n
  6. The second if<\/code> statement checks if the line, with the newline character removed using the replace()<\/code> method, is not in the lines_seen<\/code> list. If the line is not empty and has not already been seen, it is appended to the output<\/code> string and added to the lines_seen<\/code> list.<\/li>\n\n\n\n
  7. After the loop has finished, the function checks if the last character of output<\/code> is a newline character, and if so, removes it from the output<\/code> string.<\/li>\n\n\n\n
  8. The function reopens the file with write and read permissions using open()<\/code> function with ‘w+’ argument. The encoding is specified as ‘utf-8’.<\/li>\n\n\n\n
  9. The updated content of the file, which is stored in the output<\/code> string, is written to the file object using the write()<\/code> method.<\/li>\n\n\n\n
  10. Finally, the function closes the file object with the close()<\/code> method.<\/li>\n<\/ol>\n\n\n\n
    In summary, this function takes a text file, reads its content, removes any empty lines and duplicate lines, and overwrites the file with the updated content that does not contain any empty lines or duplicate lines.<\/p>\n","protected":false},"excerpt":{"rendered":"
    The objective of this Python function is to read the contents of a text file, remove any empty lines and any duplicate lines, and then overwrite the file with the updated content that no longer contains empty lines or duplicate lines. Ready-to-use Python function to empty or blank lines and duplicate lines in a file: […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_uag_custom_page_level_css":"","_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[12,17],"tags":[14,13,16,15,21],"class_list":["post-476","post","type-post","status-publish","format-standard","hentry","category-python","category-useful-function","tag-programming-language","tag-python","tag-python-code","tag-python-sample-code","tag-text-file"],"aioseo_notices":[],"yoast_head":"\nRemove empty and duplicates lines in a file - tinyytopic.com<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/tinyytopic.com\/index.php\/2023\/02\/16\/remove-empty-and-duplicates-lines-in-a-file\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Remove empty and duplicates lines in a file - tinyytopic.com\" \/>\n<meta property=\"og:description\" content=\"The objective of this Python function is to read the contents of a text file, remove any empty lines and any duplicate lines, and then overwrite the file with the updated content that no longer contains empty lines or duplicate lines. Ready-to-use Python function to empty or blank lines and duplicate lines in a file: […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/tinyytopic.com\/index.php\/2023\/02\/16\/remove-empty-and-duplicates-lines-in-a-file\/\" \/>\n<meta property=\"og:site_name\" content=\"tinyytopic.com\" \/>\n<meta property=\"article:published_time\" content=\"2023-02-16T10:56:15+00:00\" \/>\n<meta name=\"author\" content=\"tinyytopic.com\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"tinyytopic.com\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/tinyytopic.com\/index.php\/2023\/02\/16\/remove-empty-and-duplicates-lines-in-a-file\/\",\"url\":\"https:\/\/tinyytopic.com\/index.php\/2023\/02\/16\/remove-empty-and-duplicates-lines-in-a-file\/\",\"name\":\"Remove empty and duplicates lines in a file - tinyytopic.com\",\"isPartOf\":{\"@id\":\"https:\/\/tinyytopic.com\/#website\"},\"datePublished\":\"2023-02-16T10:56:15+00:00\",\"dateModified\":\"2023-02-16T10:56:15+00:00\",\"author\":{\"@id\":\"https:\/\/tinyytopic.com\/#\/schema\/person\/56c840cea8539fb221a03c5fa2ef32eb\"},\"breadcrumb\":{\"@id\":\"https:\/\/tinyytopic.com\/index.php\/2023\/02\/16\/remove-empty-and-duplicates-lines-in-a-file\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/tinyytopic.com\/index.php\/2023\/02\/16\/remove-empty-and-duplicates-lines-in-a-file\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/tinyytopic.com\/index.php\/2023\/02\/16\/remove-empty-and-duplicates-lines-in-a-file\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/tinyytopic.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Remove empty and duplicates lines in a file\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/tinyytopic.com\/#website\",\"url\":\"https:\/\/tinyytopic.com\/\",\"name\":\"tinyytopic.com\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/tinyytopic.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/tinyytopic.com\/#\/schema\/person\/56c840cea8539fb221a03c5fa2ef32eb\",\"name\":\"tinyytopic.com\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/tinyytopic.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/5f153681c8ca1e6d7287d858de51f968bb687221c89cf96d763ead4393881029?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/5f153681c8ca1e6d7287d858de51f968bb687221c89cf96d763ead4393881029?s=96&d=mm&r=g\",\"caption\":\"tinyytopic.com\"},\"sameAs\":[\"http:\/\/tinyytopic.com\"],\"url\":\"https:\/\/tinyytopic.com\/index.php\/author\/mmkmuthukumar21gmail-com\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Remove empty and duplicates lines in a file - tinyytopic.com","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/tinyytopic.com\/index.php\/2023\/02\/16\/remove-empty-and-duplicates-lines-in-a-file\/","og_locale":"en_US","og_type":"article","og_title":"Remove empty and duplicates lines in a file - tinyytopic.com","og_description":"The objective of this Python function is to read the contents of a text file, remove any empty lines and any duplicate lines, and then overwrite the file with the updated content that no longer contains empty lines or duplicate lines. Ready-to-use Python function to empty or blank lines and duplicate lines in a file: […]","og_url":"https:\/\/tinyytopic.com\/index.php\/2023\/02\/16\/remove-empty-and-duplicates-lines-in-a-file\/","og_site_name":"tinyytopic.com","article_published_time":"2023-02-16T10:56:15+00:00","author":"tinyytopic.com","twitter_card":"summary_large_image","twitter_misc":{"Written by":"tinyytopic.com","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/tinyytopic.com\/index.php\/2023\/02\/16\/remove-empty-and-duplicates-lines-in-a-file\/","url":"https:\/\/tinyytopic.com\/index.php\/2023\/02\/16\/remove-empty-and-duplicates-lines-in-a-file\/","name":"Remove empty and duplicates lines in a file - tinyytopic.com","isPartOf":{"@id":"https:\/\/tinyytopic.com\/#website"},"datePublished":"2023-02-16T10:56:15+00:00","dateModified":"2023-02-16T10:56:15+00:00","author":{"@id":"https:\/\/tinyytopic.com\/#\/schema\/person\/56c840cea8539fb221a03c5fa2ef32eb"},"breadcrumb":{"@id":"https:\/\/tinyytopic.com\/index.php\/2023\/02\/16\/remove-empty-and-duplicates-lines-in-a-file\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/tinyytopic.com\/index.php\/2023\/02\/16\/remove-empty-and-duplicates-lines-in-a-file\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/tinyytopic.com\/index.php\/2023\/02\/16\/remove-empty-and-duplicates-lines-in-a-file\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/tinyytopic.com\/"},{"@type":"ListItem","position":2,"name":"Remove empty and duplicates lines in a file"}]},{"@type":"WebSite","@id":"https:\/\/tinyytopic.com\/#website","url":"https:\/\/tinyytopic.com\/","name":"tinyytopic.com","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/tinyytopic.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/tinyytopic.com\/#\/schema\/person\/56c840cea8539fb221a03c5fa2ef32eb","name":"tinyytopic.com","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/tinyytopic.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/5f153681c8ca1e6d7287d858de51f968bb687221c89cf96d763ead4393881029?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/5f153681c8ca1e6d7287d858de51f968bb687221c89cf96d763ead4393881029?s=96&d=mm&r=g","caption":"tinyytopic.com"},"sameAs":["http:\/\/tinyytopic.com"],"url":"https:\/\/tinyytopic.com\/index.php\/author\/mmkmuthukumar21gmail-com\/"}]}},"uagb_featured_image_src":{"full":false,"thumbnail":false,"medium":false,"medium_large":false,"large":false,"1536x1536":false,"2048x2048":false},"uagb_author_info":{"display_name":"tinyytopic.com","author_link":"https:\/\/tinyytopic.com\/index.php\/author\/mmkmuthukumar21gmail-com\/"},"uagb_comment_info":96,"uagb_excerpt":"The objective of this Python function is to read the contents of a text file, remove any empty lines and any duplicate lines, and then overwrite the file with the updated content that no longer contains empty lines or duplicate lines. Ready-to-use Python function to empty or blank lines and duplicate lines in a file:…","_links":{"self":[{"href":"https:\/\/tinyytopic.com\/index.php\/wp-json\/wp\/v2\/posts\/476","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tinyytopic.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tinyytopic.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tinyytopic.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tinyytopic.com\/index.php\/wp-json\/wp\/v2\/comments?post=476"}],"version-history":[{"count":4,"href":"https:\/\/tinyytopic.com\/index.php\/wp-json\/wp\/v2\/posts\/476\/revisions"}],"predecessor-version":[{"id":479,"href":"https:\/\/tinyytopic.com\/index.php\/wp-json\/wp\/v2\/posts\/476\/revisions\/479"}],"wp:attachment":[{"href":"https:\/\/tinyytopic.com\/index.php\/wp-json\/wp\/v2\/media?parent=476"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tinyytopic.com\/index.php\/wp-json\/wp\/v2\/categories?post=476"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tinyytopic.com\/index.php\/wp-json\/wp\/v2\/tags?post=476"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}