# UTF8 everywhere

**URL:** https://discourse.paraview.org/t/utf8-everywhere/3385
**Category:** Development
**Created:** [January 23, 2020, 9:33am UTC](https://discourse.paraview.org/t/utf8-everywhere/3385 "2020-01-23T09:33:09Z")
**Posts on this page:** 11
**Page:** 1

<div class="post-metadata">

### Author: ![todoooo](https://discourse.paraview.org/user_avatar/discourse.paraview.org/todoooo/32/2819_2.png) [@todoooo](https://discourse.paraview.org/u/todoooo)
#### Post date: [January 23, 2020, 9:33am UTC](https://discourse.paraview.org/t/utf8-everywhere/3385/1 "2020-01-23T09:33:09Z")

</div>

I have been working on several merge requests in VTK, [https://gitlab.kitware.com/vtk/vtk/merge\_requests/6122](https://gitlab.kitware.com/vtk/vtk/merge_requests/6122) [https://gitlab.kitware.com/vtk/vtk/merge\_requests/6291](https://gitlab.kitware.com/vtk/vtk/merge_requests/6291) & [https://gitlab.kitware.com/vtk/vtk/merge\_requests/6301](https://gitlab.kitware.com/vtk/vtk/merge_requests/6301), which introduce the constraint that all file and path names (and eventually all string data) passed across the VTK API are utf-8 encoded.

It has come to my attention that some Paraview users make use of extended character sets with non-utf8 locales, so I’m wondering how many people would object to this new requirement?

---

<div class="post-metadata">

### Author: ![mwestphal](https://discourse.paraview.org/user_avatar/discourse.paraview.org/mwestphal/32/17_2.png) [@mwestphal](https://discourse.paraview.org/u/mwestphal)
#### Post date: [January 23, 2020, 9:38am UTC](https://discourse.paraview.org/t/utf8-everywhere/3385/2 "2020-01-23T09:38:02Z")

</div>

> which introduce the constraint that all file and path names (and eventually all string data) passed to/from VTK are utf-8 encoded.

Once again, it is not clear to me where, in the code, is this requirement present. could you precise that ?

> so I’m wondering how many people would object to this new requirement ?

Anyone without a UTF-8 locale will be impacted by the proposed change to ParaView.  
Type `locale -a` in a terminal on linux to know.  
On windows, the info is in the language settings.

---

<div class="post-metadata">

### Author: ![todoooo](https://discourse.paraview.org/user_avatar/discourse.paraview.org/todoooo/32/2819_2.png) [@todoooo](https://discourse.paraview.org/u/todoooo)
#### Post date: [January 23, 2020, 10:03am UTC](https://discourse.paraview.org/t/utf8-everywhere/3385/3 "2020-01-23T10:03:28Z")

</div>

Anyone using ANSI (128 char) file and path names will not be affected, regardless of their locale settings, since utf-8 encoding is identical to plain English text.

---

<div class="post-metadata">

### Author: ![mwestphal](https://discourse.paraview.org/user_avatar/discourse.paraview.org/mwestphal/32/17_2.png) [@mwestphal](https://discourse.paraview.org/u/mwestphal)
#### Post date: [January 23, 2020, 10:05am UTC](https://discourse.paraview.org/t/utf8-everywhere/3385/4 "2020-01-23T10:05:25Z")

</div>

Why do you say that all file and path name have to be UTF-8 ?

---

<div class="post-metadata">

### Author: ![todoooo](https://discourse.paraview.org/user_avatar/discourse.paraview.org/todoooo/32/2819_2.png) [@todoooo](https://discourse.paraview.org/u/todoooo)
#### Post date: [January 23, 2020, 10:14am UTC](https://discourse.paraview.org/t/utf8-everywhere/3385/5 "2020-01-23T10:14:40Z")

</div>

The requirement comes about due to the introduction of vtksys for opening files/streams and as a result of this change in MR 6122  
 ![image](https://discourse.paraview.org/uploads/default/original/2X/4/462e3c3428c12168675d5e88be33ba1598f6307c.png)

---

<div class="post-metadata">

### Author: ![mwestphal](https://discourse.paraview.org/user_avatar/discourse.paraview.org/mwestphal/32/17_2.png) [@mwestphal](https://discourse.paraview.org/u/mwestphal)
#### Post date: [January 23, 2020, 10:23am UTC](https://discourse.paraview.org/t/utf8-everywhere/3385/6 "2020-01-23T10:23:20Z")

</div>

Much clearer now. Thanks.

There is indeed no way around it. However it means **we drop support of non-UTF8 locale on windows** , so we must be clear on that beforehand.

@utkarsh.ayachit @cory.quammen : thoughts ?

---

<div class="post-metadata">

### Author: ![cory.quammen](https://discourse.paraview.org/user_avatar/discourse.paraview.org/cory.quammen/32/11193_2.png) [@cory.quammen](https://discourse.paraview.org/u/cory.quammen)
#### Post date: [January 23, 2020, 6:56pm UTC](https://discourse.paraview.org/t/utf8-everywhere/3385/7 "2020-01-23T18:56:28Z")

</div>

Afraid I don’t know enough about character encodings to contribute. Are the limitations on non-UTF8 locales likely to affect many Windows users?

---

<div class="post-metadata">

### Author: ![todoooo](https://discourse.paraview.org/user_avatar/discourse.paraview.org/todoooo/32/2819_2.png) [@todoooo](https://discourse.paraview.org/u/todoooo)
#### Post date: [January 23, 2020, 8:27pm UTC](https://discourse.paraview.org/t/utf8-everywhere/3385/8 "2020-01-23T20:27:07Z")

</div>

The purpose of the changes is actually to allow text in any language to be treated consistently and reliably by VTK regardless of the user locale.

---

<div class="post-metadata">

### Author: ![mwestphal](https://discourse.paraview.org/user_avatar/discourse.paraview.org/mwestphal/32/17_2.png) [@mwestphal](https://discourse.paraview.org/u/mwestphal)
#### Post date: [January 24, 2020, 4:14am UTC](https://discourse.paraview.org/t/utf8-everywhere/3385/9 "2020-01-24T04:14:14Z")

</div>

Indeed, all these changes will indeed improve support for UTF-8 encodings greatly.

However, all Chinese user may be impacted by this, see here :

> **[GB 18030](https://en.wikipedia.org/wiki/GB_18030)**
>
> GB 18030 is a Chinese government standard, described as Information Technology — Chinese coded character set and defines the required language and character support necessary for software in China. GB18030 is the registered Internet name for the official character set of the People's Republic of China (PRC) superseding GB2312. As a Unicode Transformation Format\[a\] (i.e. an encoding of all Unicode code points), GB18030 supports both simplified and traditional Chinese characters. It is also comp In...

---

<div class="post-metadata">

### Author: ![todoooo](https://discourse.paraview.org/user_avatar/discourse.paraview.org/todoooo/32/2819_2.png) [@todoooo](https://discourse.paraview.org/u/todoooo)
#### Post date: [January 24, 2020, 4:20am UTC](https://discourse.paraview.org/t/utf8-everywhere/3385/10 "2020-01-24T04:20:42Z")

</div>

I don’t see the problem. Every character from every language can be encoded in UTF-8.

 ![image](https://discourse.paraview.org/uploads/default/original/2X/2/2097d5a34e6d83faa1ee224e53c4e892750517e3.png)

---

<div class="post-metadata">

### Author: ![mwestphal](https://discourse.paraview.org/user_avatar/discourse.paraview.org/mwestphal/32/17_2.png) [@mwestphal](https://discourse.paraview.org/u/mwestphal)
#### Post date: [January 24, 2020, 4:22am UTC](https://discourse.paraview.org/t/utf8-everywhere/3385/11 "2020-01-24T04:22:17Z")

</div>

Great ! I missed this part ! GB232, which was historically used before GB 18030 did not support it.  
There still must be user in China using GB232 but that may be fine to tell them to upgrade.
