# Paraview doesn't read vtk xml with appended data written as base64

**URL:** https://discourse.paraview.org/t/paraview-doesnt-read-vtk-xml-with-appended-data-written-as-base64/9203
**Category:** ParaView Support
**Tags:** vtk, python
**Created:** [March 16, 2022, 11:27pm UTC](https://discourse.paraview.org/t/paraview-doesnt-read-vtk-xml-with-appended-data-written-as-base64/9203 "2022-03-16T23:27:46Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![jpmorr](https://discourse.paraview.org/user_avatar/discourse.paraview.org/jpmorr/32/8745_2.png) [@jpmorr](https://discourse.paraview.org/u/jpmorr)
#### Post date: [March 16, 2022, 11:27pm UTC](https://discourse.paraview.org/t/paraview-doesnt-read-vtk-xml-with-appended-data-written-as-base64/9203/1 "2022-03-16T23:27:46Z")

</div>

Hi,

I’m using `python` to create my `VTK` files in an `XML` format and I’ve managed to get everything working for ascii files, binary files and appended files with raw encoding. However I am unable to get the appended data to work and always seem to get the following error in paraview

```auto
Error reading uncompressed binary data header. Read 7 of 8 bytes.
ERROR: In ./VTK/IO/XMLParser/vtkXMLDataParser.cxx, line 653
vtkXMLDataParser (0x55f909190850): Error reading uncompressed binary data header. Read 7 of 8 bytes.

```

For the following examples I’m just generating random data (with a seed) in python:

```auto
    np.random.seed(77)

    npoints = 20
    x = np.random.rand(npoints)
    y = np.random.rand(npoints)
    z = np.random.rand(npoints)
    ID = np.arange(npoints) + 1
    pressure = np.random.rand(npoints)
    temp = np.random.rand(npoints)
    force = np.random.rand(npoints, 3)

    pos = np.vstack([x, y, z]).T

```

The `ascii` file I generate is this:

```auto
<?xml version="1.0"?>
<VTKFile type="PolyData" version="1.0" byte_order="LittleEndian" header_type="UInt64">
  <PolyData>
    <Piece NumberOfPoints="20" NumberOfVerts="20" NumberOfLines="0" NumberOfStrips="0" NumberOfPolys="0">
      <PointData Scalars="1_temp" Vectors="3_force">
        <DataArray Name="1_temp" NumberOfComponents="1" type="Float64" format="ascii">
          0.4204270035661765 0.9490145030363609 0.7864711816566549 0.4815468959885630 0.0119215535827562 0.0597207807092479
          0.8257177483439162 0.4046369872739349 0.0531531123171690 0.2596390482071086 0.2811506610562717 0.9433030410624577
          0.8936293596271191 0.2022892694899581 0.1003608087345019 0.5730430655359530 0.7453773741488717 0.6007604785160442
          0.9092395428890321 0.9385777624120453
        </DataArray>
        <DataArray Name="2_pressure" NumberOfComponents="1" type="Float64" format="ascii">
          0.5890720970505128 0.3268986086578718 0.1448144482177102 0.2576384903723663 0.8574458883976873 0.6839737520406283
          0.9986326035811655 0.3194556656082242 0.4534082266242594 0.8676221777748950 0.6385221808378060 0.3318923019829476
          0.5419776327479333 0.1123238088406830 0.8165234436255825 0.2209316463478144 0.5149742681090496 0.5260877679247680
          0.2770886884732112 0.9044206316605501
        </DataArray>
        <DataArray Name="3_force" NumberOfComponents="3" type="Float64" format="ascii">
          0.5615661006777212 0.8908316333576722 0.3124782869770458 0.7710535361311034 0.2136507101264129 0.2883443963127228
          0.2734427951922838 0.2485007122242413 0.0144745384478043 0.6091129763543156 0.8618458103278591 0.2591405994003485
          0.8668647914278148 0.4265298541907321 0.1603909008190777 0.0565316315409430 0.4228071934122273 0.7359960547933326
          0.4769537916279307 0.7006992708984805 0.9261806156102312 0.4526842850674920 0.3917947827194505 0.3592419246788816
          0.9207452697946850 0.0278180708490150 0.2349904981777623 0.0795443510228309 0.5657883468686661 0.2948885661324933
          0.7193769267541885 0.0988020129816635 0.0268196366997491 0.7018310907399513 0.7914683737300843 0.3782688291338249
          0.2473226108228560 0.2627823218786545 0.4984137186921088 0.3041873590603215 0.7186392108467865 0.6106608253770643
          0.0646729072148587 0.1480143523868742 0.9702434112898988 0.4715222037958284 0.7331722522876482 0.2106973939969884
          0.6426786141941642 0.9115921673564267 0.0345287813667512 0.1773567248506672 0.2962514934470256 0.3126333475071587
          0.3249249442272334 0.5780774130897147 0.9983199556809254 0.3547831120917448 0.0207988496596299 0.0996248310145578
        </DataArray>
      </PointData>
      <CellData>
      </CellData>
      <Points>
        <DataArray Name="Points" NumberOfComponents="3" type="Float64" format="ascii">
          0.9191090317991818 0.4523130130592968 0.4455559100158843 0.6421955999238554 0.1757747380313244 0.9770526614339072
          0.7537122297291767 0.0493769993144870 0.7293940090533850 0.1393145681027350 0.2924753386949548 0.3122350583311531
          0.0873195479896001 0.0667991310914613 0.8947552392940534 0.7880020580364918 0.7511564859911245 0.7832735951745574
          0.3261509370344888 0.0637715240806562 0.2620003413345412 0.5410678214759677 0.4319083194117714 0.3094831861549427
          0.2402351758680060 0.3641724134430167 0.1294506297724709 0.5454229255759049 0.1519715256925609 0.4221713562335928
          0.4005545036798892 0.5467103397159158 0.9397650349673574 0.7151918872797569 0.4432930376023336 0.3670428731605497
          0.8366799410811948 0.0360613100029110 0.4347749726828846 0.5884811425466829 0.8228931912295053 0.9170935543593193
          0.2961545640333596 0.2732926831299841 0.9472939210005341 0.2810176906539642 0.1689852192728417 0.2547729480887090
          0.7055972446574292 0.6443697547109174 0.7097035779964328 0.4225964301897521 0.1075410788025926 0.0499764048396870
          0.0573159852472062 0.3532450981171027 0.1347439306209732 0.7470273067071216 0.3857036581156064 0.5065059431067037
        </DataArray>
      </Points>
      <Verts>
        <DataArray Name="connectivity" NumberOfComponents="1" type="Int32" format="ascii">
          0 1 2 3 4 5
          6 7 8 9 10 11
          12 13 14 15 16 17
          18 19
        </DataArray>
        <DataArray Name="offsets" NumberOfComponents="1" type="Int32" format="ascii">
          1 2 3 4 5 6
          7 8 9 10 11 12
          13 14 15 16 17 18
          19 20
        </DataArray>
      </Verts>
      <Lines>
        <DataArray Name="connectivity" NumberOfComponents="1" type="Float64" format="ascii">
        </DataArray>
        <DataArray Name="offsets" NumberOfComponents="1" type="Float64" format="ascii">
        </DataArray>
      </Lines>
      <Strips>
        <DataArray Name="connectivity" NumberOfComponents="1" type="Float64" format="ascii">
        </DataArray>
        <DataArray Name="offsets" NumberOfComponents="1" type="Float64" format="ascii">
        </DataArray>
      </Strips>
      <Polys>
        <DataArray Name="connectivity" NumberOfComponents="1" type="Float64" format="ascii">
        </DataArray>
        <DataArray Name="offsets" NumberOfComponents="1" type="Float64" format="ascii">
        </DataArray>
      </Polys>
    </Piece>
  </PolyData>
</VTKFile>

```

And the binary file generated is this:

```auto
<?xml version="1.0"?>
<VTKFile type="PolyData" version="1.0" byte_order="LittleEndian" header_type="UInt64">
  <PolyData>
    <Piece NumberOfPoints="20" NumberOfVerts="20" NumberOfLines="0" NumberOfStrips="0" NumberOfPolys="0">
      <PointData Scalars="1_temp" Vectors="3_force">
        <DataArray Name="1_temp" NumberOfComponents="1" type="Float64" format="binary">
          oAAAAAAAAAAaAqupRujaP08Rv6lTXu4/IceOnMUq6T9et3ASqtHePwApC9ZTaog/gFsT4LiTrj/kpJugR2zqPzb8xYiS5dk/EGwqfuI2qz8yGjQZ7Z3QP+gRn1df/tE/SJTy24kv7j85AUuZnJjsP8hCZWKd5Mk/iJZQ9z6xuT/ppTVpXlbiP3O6pKYh2uc/Q5X+CW454z86B56GfRjtP8IASjvUCO4/
        </DataArray>
        <DataArray Name="2_pressure" NumberOfComponents="1" type="Float64" format="binary">
          oAAAAAAAAADMLfq5rdniPxTIUiTo69Q/PKuKo0eJwj+YxJUmJn3QP1k/GFwycOs/+8cK7Bzj5T/KNqNczPTvP4LKEy32cdQ/0KpF8KME3T+netqVj8PrP9cBjxHGbuQ/DOazNbk91T+gG/p54VfhP3BriM1Awbw/70DZxfUg6j/4jtv5fEfMP7vq+VCreuA/IgDCA7bV4D/WX8Ux0rvRP0TpWYkD8ew/
        </DataArray>
        <DataArray Name="3_force" NumberOfComponents="3" type="Float64" format="binary">
          4AEAAAAAAABvf554WfjhP2m+JHd4rOg/MF+rNRaA0T8rWiB/2n3jP/uPJztbves/oNRQybbxrD+4sC8yaYbePxLL8YHH+Nw/NOC2yL527T+QfznCBF20P9D/vMIiBec/Q2nBeWZ15j8E04VuRKjPP/xSwUHOd9M/yNBsVWeOsD8ABSl3ay3eP2Guub3SkOQ/5Ml6CqCzxj/uE0f+kcvUP3zb5jnEtNY/Tm5wV7GB7D+YS2EO6FjLP+CVBBDfzs8/U7Axqj2U6z/0E6HfQ0zbPyAFDudFD9s/buGa4CBs5j92gKFsKhPZP4AnIyJXfJw/S0jIKfAa4j+oA4m2Fku5Pwimn3u1U+k/mOOb8WzR0D/uuYaoF//mPyCNa2Ei8sI/nx+9pyV25z9UvUJWwyvtP6bA79LI9dI/99n4M5x/4j8Ag1QsS0yVP2qy0e2k/9M/9HgJDjx00j/AaRCq06SNP6Jh33PCldA/hAXMZLCHxD8AUSqZR43nP9yFx4dFo+0/3EN219H91j+MqUUsKxTOP5I34Ep039I/gGgQWpt2mz+edo52jjXYPzJDaqcC5t8/5Ss+koiK4z92yRTpOwzvPxzlSNgh+Mo/QFt7pcGtoT/6qctMLwLUP/P/ErE88u8/sLkTTwOBuT8=
        </DataArray>
      </PointData>
      <CellData>
      </CellData>
      <Points>
        <DataArray Name="Points" NumberOfComponents="3" type="Float64" format="binary">
          4AEAAAAAAACJJCFYV2ntPw5GqUey8tw/DLKD7/yD3D+f22nJ3YzkP1hcp1/Jf8Y/MIJq8QNE7z+UBikcaR7oP6DudSrxR6k/MgvZGjJX5z84xu1MD9XBP+g0pXvqt9I/6JoMwaj70z+wterqklq2PyBlcHO/GbE/5ClWvdWh7D9RUY4XUDfpPz9TsFN5Ceg/ywpjyZMQ6T9Y3QcuqN/UP8CtV6JUU7A/+qZkFJ3E0D8sCMV2bVDhP7ybr8pipNs/iKHMkJLO0z+s19m3BsDOPyD2dc+ZTtc/AOGoltaRwD/2yXrHGnThPzjqYo7Nc8M/JjMVAtsE2z9MhWRbr6LZP4fgrq6mfuE/uH1jH44S7j/2ZMcY2uLmPzT6wsLpXtw/gBUdZKF91z/LNgUDFcbqP0ASOsagdqI/wrQyaFrT2z96n7Fn1tTiP03RDRokVeo/xA/rlNRY7T8YZ8VFMvTSPz7hEZigfdE/a7NMVztQ7j+6Mr2fMfzRPxxiJMNOocU/opP8MTNO0D+ecT6sQJTmP1N64FGtnuQ/5SkrR+S15j+2kMTl0QvbP5AGb+jPh7s/cFq34IGWqT9Qe1lUhVitPzio+FORm9Y/WACtA0o/wT9k1XDPpefnP0h2Y2Ver9g/Ce+b80s14D8=
        </DataArray>
      </Points>
      <Verts>
        <DataArray Name="connectivity" NumberOfComponents="1" type="Int32" format="binary">
          UAAAAAAAAAAAAAAAAQAAAAIAAAADAAAABAAAAAUAAAAGAAAABwAAAAgAAAAJAAAACgAAAAsAAAAMAAAADQAAAA4AAAAPAAAAEAAAABEAAAASAAAAEwAAAA==
        </DataArray>
        <DataArray Name="offsets" NumberOfComponents="1" type="Int32" format="binary">
          UAAAAAAAAAABAAAAAgAAAAMAAAAEAAAABQAAAAYAAAAHAAAACAAAAAkAAAAKAAAACwAAAAwAAAANAAAADgAAAA8AAAAQAAAAEQAAABIAAAATAAAAFAAAAA==
        </DataArray>
      </Verts>
      <Lines>
        <DataArray Name="connectivity" NumberOfComponents="1" type="Float64" format="binary">
        </DataArray>
        <DataArray Name="offsets" NumberOfComponents="1" type="Float64" format="binary">
        </DataArray>
      </Lines>
      <Strips>
        <DataArray Name="connectivity" NumberOfComponents="1" type="Float64" format="binary">
        </DataArray>
        <DataArray Name="offsets" NumberOfComponents="1" type="Float64" format="binary">
        </DataArray>
      </Strips>
      <Polys>
        <DataArray Name="connectivity" NumberOfComponents="1" type="Float64" format="binary">
        </DataArray>
        <DataArray Name="offsets" NumberOfComponents="1" type="Float64" format="binary">
        </DataArray>
      </Polys>
    </Piece>
  </PolyData>
</VTKFile>

```

Both of these work fine in Paraview with no errors.

I also successfully create the appended data using the raw formatting but when I create the append file with `base64` encoding paraview fails to read this file:

```auto
<?xml version="1.0"?>
<VTKFile type="PolyData" version="1.0" byte_order="LittleEndian" header_type="UInt64">
  <PolyData>
    <Piece NumberOfPoints="20" NumberOfVerts="20" NumberOfLines="0" NumberOfStrips="0" NumberOfPolys="0">
      <PointData Scalars="1_temp" Vectors="3_force">
        <DataArray Name="1_temp" type="Float64" format="appended" NumberOfComponents="1" offset="0"/>
        <DataArray Name="2_pressure" type="Float64" format="appended" NumberOfComponents="1" offset="168"/>
        <DataArray Name="3_force" type="Float64" format="appended" NumberOfComponents="3" offset="336"/>
      </PointData>
      <CellData>
      </CellData>
      <Points>
        <DataArray Name="Points" type="Float64" format="appended" NumberOfComponents="3" offset="824"/>
      </Points>
      <Verts>
        <DataArray Name="connectivity" type="Int32" format="appended" NumberOfComponents="1" offset="1312"/>
        <DataArray Name="offsets" type="Int32" format="appended" NumberOfComponents="1" offset="1400"/>
      </Verts>
      <Lines>
        <DataArray Name="connectivity" type="Float64" format="appended" NumberOfComponents="1" offset="1488"/>
        <DataArray Name="offsets" type="Float64" format="appended" NumberOfComponents="1" offset="1496"/>
      </Lines>
      <Strips>
        <DataArray Name="connectivity" type="Float64" format="appended" NumberOfComponents="1" offset="1504"/>
        <DataArray Name="offsets" type="Float64" format="appended" NumberOfComponents="1" offset="1512"/>
      </Strips>
      <Polys>
        <DataArray Name="connectivity" type="Float64" format="appended" NumberOfComponents="1" offset="1520"/>
        <DataArray Name="offsets" type="Float64" format="appended" NumberOfComponents="1" offset="1528"/>
      </Polys>
    </Piece>
  </PolyData>
  <AppendedData encoding="base64">
    _oAAAAAAAAA==GgKrqUbo2j9PEb+pU17uPyHHjpzFKuk/XrdwEqrR3j8AKQvWU2qIP4BbE+C4k64/5KSboEds6j82/MWIkuXZPxBsKn7iNqs/Mho0Ge2d0D/oEZ9XX/7RP0iU8tuJL+4/OQFLmZyY7D/IQmVineTJP4iWUPc+sbk/6aU1aV5W4j9zuqSmIdrnP0OV/gluOeM/Ogeehn0Y7T/CAEo71AjuPw==oAAAAAAAAA==zC36ua3Z4j8UyFIk6OvUPzyriqNHicI/mMSVJiZ90D9ZPxhcMnDrP/vHCuwc4+U/yjajXMz07z+CyhMt9nHUP9CqRfCjBN0/p3ralY/D6z/XAY8Rxm7kPwzmszW5PdU/oBv6eeFX4T9wa4jNQMG8P+9A2cX1IOo/+I7b+XxHzD+76vlQq3rgPyIAwgO21eA/1l/FMdK70T9E6VmJA/HsPw==4AEAAAAAAA==b3+eeFn44T9pviR3eKzoPzBfqzUWgNE/K1ogf9p94z/7jyc7W73rP6DUUMm28aw/uLAvMmmG3j8Sy/GBx/jcPzTgtsi+du0/kH85wgRdtD/Q/7zCIgXnP0NpwXlmdeY/BNOFbkSozz/8UsFBznfTP8jQbFVnjrA/AAUpd2st3j9hrrm90pDkP+TJegqgs8Y/7hNH/pHL1D982+Y5xLTWP05ucFexgew/mEthDuhYyz/glQQQ387PP1OwMao9lOs/9BOh30NM2z8gBQ7nRQ/bP27hmuAgbOY/doChbCoT2T+AJyMiV3ycP0tIyCnwGuI/qAOJthZLuT8Ipp97tVPpP5jjm/Fs0dA/7rmGqBf/5j8gjWthIvLCP58fvaclduc/VL1CVsMr7T+mwO/SyPXSP/fZ+DOcf+I/AINULEtMlT9qstHtpP/TP/R4CQ48dNI/wGkQqtOkjT+iYd9zwpXQP4QFzGSwh8Q/AFEqmUeN5z/chceHRaPtP9xDdtfR/dY/jKlFLCsUzj+SN+BKdN/SP4BoEFqbdps/nnaOdo412D8yQ2qnAubfP+UrPpKIiuM/dskU6TsM7z8c5UjYIfjKP0Bbe6XBraE/+qnLTC8C1D/z/xKxPPLvP7C5E08Dgbk/4AEAAAAAAA==iSQhWFdp7T8ORqlHsvLcPwyyg+/8g9w/n9tpyd2M5D9YXKdfyX/GPzCCavEDRO8/lAYpHGke6D+g7nUq8UepPzIL2RoyV+c/OMbtTA/VwT/oNKV76rfSP+iaDMGo+9M/sLXq6pJatj8gZXBzvxmxP+QpVr3Voew/UVGOF1A36T8/U7BTeQnoP8sKY8mTEOk/WN0HLqjf1D/ArVeiVFOwP/qmZBSdxNA/LAjFdm1Q4T+8m6/KYqTbP4ihzJCSztM/rNfZtwbAzj8g9nXPmU7XPwDhqJbWkcA/9sl6xxp04T846mKOzXPDPyYzFQLbBNs/TIVkW6+i2T+H4K6upn7hP7h9Yx+OEu4/9mTHGNri5j80+sLC6V7cP4AVHWShfdc/yzYFAxXG6j9AEjrGoHaiP8K0Mmha09s/ep+xZ9bU4j9N0Q0aJFXqP8QP65TUWO0/GGfFRTL00j8+4RGYoH3RP2uzTFc7UO4/ujK9nzH80T8cYiTDTqHFP6KT/DEzTtA/nnE+rECU5j9TeuBRrZ7kP+UpK0fkteY/tpDE5dEL2z+QBm/oz4e7P3Bat+CBlqk/UHtZVIVYrT84qPhTkZvWP1gArQNKP8E/ZNVwz6Xn5z9IdmNlXq/YPwnvm/NLNeA/UAAAAAAAAA==AAAAAAEAAAACAAAAAwAAAAQAAAAFAAAABgAAAAcAAAAIAAAACQAAAAoAAAALAAAADAAAAA0AAAAOAAAADwAAABAAAAARAAAAEgAAABMAAAA=UAAAAAAAAA==AQAAAAIAAAADAAAABAAAAAUAAAAGAAAABwAAAAgAAAAJAAAACgAAAAsAAAAMAAAADQAAAA4AAAAPAAAAEAAAABEAAAASAAAAEwAAABQAAAA=AAAAAAAAAA==AAAAAAAAAA==AAAAAAAAAA==AAAAAAAAAA==AAAAAAAAAA==AAAAAAAAAA==
  </AppendedData>
</VTKFile>

```

There is almost zero information on this in the VTK documentation, just that you can use it.

> The appended data section begins with the first character after the underscore inside the _AppendedData_ element. The underscore is not part of the data, but is always present. Data in this section is always in binary form, but can be compressed and/or base64 encoded. The byte-order of the data matches that specified by the byte\_order attribute of the _VTKFile_ element. Each _DataArray_’s data are stored contiguously and appended immediately after the previous _DataArray_’s data without a seperator. The _DataArray_’s _offset_ attribute indicates the file position offset from the first character after the underscore to the beginning its data.

The only information I could find that suggests how it should be written is at [this post](https://mathema.tician.de/what-they-dont-tell-you-about-vtk-xml-binary-formats/) - the key being that the block size needs to written separately from the data, but this seems to be the opposite of the inline binary encoding.

Does anyone know why the `base64` appended data doesn’t work?

---

<div class="post-metadata">

### Author: ![jpmorr](https://discourse.paraview.org/user_avatar/discourse.paraview.org/jpmorr/32/8745_2.png) [@jpmorr](https://discourse.paraview.org/u/jpmorr)
#### Post date: [June 10, 2024, 9:14am UTC](https://discourse.paraview.org/t/paraview-doesnt-read-vtk-xml-with-appended-data-written-as-base64/9203/2 "2024-06-10T09:14:17Z")

</div>

Closing this one myself as I’ve solved by now. There were two issues: incorrect file format and paraview bug.

I was writing the incorrect data in binary format at the end of the file (you can see the original binary strings are different from what is in the appended file. This was an easy fix.

The corrected file was still not being loaded into Paraview without throwing an error despite being read in other vtk based viewers, but this seems to have been related to another bug as mentioned in this [post](https://discourse.paraview.org/t/paraview-appended-encoded-vtu-file-not-working/14799).