Fix Python – Character reading from file in Python


Asked By – Graviton

In a text file, there is a string “I don’t like this”.

However, when I read it into a string, it becomes “I don\xe2\x80\x98t like this”. I understand that \u2018 is the unicode representation of “‘”. I use

f1 = open (file1, "r")
text =

command to do the reading.

Now, is it possible to read the string in such a way that when it is read into the string, it is “I don’t like this”, instead of “I don\xe2\x80\x98t like this like this”?

Second edit: I have seen some people use mapping to solve this problem, but really, is there no built-in conversion that does this kind of ANSI to unicode ( and vice versa) conversion?

Now we will see solution for issue: Character reading from file in Python



Reading Unicode from a file is therefore simple:

import codecs
with'unicode.rst', encoding='utf-8') as f:
    for line in f:
        print repr(line)

It’s also possible to open files in update mode, allowing both reading and writing:

with'test', encoding='utf-8', mode='w+') as f:
    f.write(u'\u4500 blah blah blah\n')
    print repr(f.readline()[:1])

EDIT: I’m assuming that your intended goal is just to be able to read the file properly into a string in Python. If you’re trying to convert to an ASCII string from Unicode, then there’s really no direct way to do so, since the Unicode characters won’t necessarily exist in ASCII.

If you’re trying to convert to an ASCII string, try one of the following:

  1. Replace the specific unicode chars with ASCII equivalents, if you are only looking to handle a few special cases such as this particular example

  2. Use the unicodedata module’s normalize() and the string.encode() method to convert as best you can to the next closest ASCII equivalent (Ref

    >>> teststr
    u'I don\xe2\x80\x98t like this'
    >>> unicodedata.normalize('NFKD', teststr).encode('ascii', 'ignore')
    'I donat like this'

This question is answered By – Jay

This answer is collected from stackoverflow and reviewed by FixPython community admins, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0