Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

struct.error: unpack requires a buffer of 8 bytes #151

Open
Dragony opened this issue May 19, 2021 · 5 comments
Open

struct.error: unpack requires a buffer of 8 bytes #151

Dragony opened this issue May 19, 2021 · 5 comments

Comments

@Dragony
Copy link

Dragony commented May 19, 2021

Hello there! Thanks for investing so much time into such a usefull tool :-)

Today I tried to extract data via the CLI:

python3 .\PyPoE\cli\exporter\core.py dat json content.json --files Mods.dat

That seems to work for a while until:

C:\Users\drago\AppData\Local\Temp\tmpe4h6a1cv\chunk38.in:    23496 =>   262144 (0.00 seconds, 311.48 MB/s)
C:\Users\drago\AppData\Local\Temp\tmpe4h6a1cv\chunk39.in:    23672 =>   262144 (0.00 seconds, 303.55 MB/s)
C:\Users\drago\AppData\Local\Temp\tmpe4h6a1cv\chunk40.in:    14561 =>   147600 (0.00 seconds, 259.27 MB/s)
  0%|                                                                                                                                                                                                                   | 0/1 [00:09<?, ?it/s]
23:43:09 Traceback (most recent call last):
  File "C:\Users\drago\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pypoe-1.0.0a0-py3.9.egg\PyPoE\cli\core.py", line 145, in run
    code = args.func(args)
  File "C:\Users\drago\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pypoe-1.0.0a0-py3.9.egg\PyPoE\cli\exporter\dat\parsers\json.py", line 109, in handle
    dat_files = self._read_dat_files(args)
  File "C:\Users\drago\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pypoe-1.0.0a0-py3.9.egg\PyPoE\cli\exporter\dat\handler.py", line 135, in _read_dat_files
    df.read(file_path_or_raw=data, use_dat_value=False)
  File "C:\Users\drago\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pypoe-1.0.0a0-py3.9.egg\PyPoE\poe\file\shared\__init__.py", line 214, in read
    return self.get_read_buffer(file_path_or_raw, self._read, *args, **kwargs)
  File "C:\Users\drago\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pypoe-1.0.0a0-py3.9.egg\PyPoE\poe\file\shared\__init__.py", line 172, in get_read_buffer
    return function(*args, buffer=BytesIO(file_path_or_raw), **kwargs)
  File "C:\Users\drago\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pypoe-1.0.0a0-py3.9.egg\PyPoE\poe\file\dat.py", line 946, in _read
    self.reader.read(buffer.read())
  File "C:\Users\drago\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pypoe-1.0.0a0-py3.9.egg\PyPoE\poe\file\dat.py", line 849, in read
    self.table_data.append(self._process_row(i))
  File "C:\Users\drago\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pypoe-1.0.0a0-py3.9.egg\PyPoE\poe\file\dat.py", line 792, in _process_row
    row_data.append(self._cast_from_spec(spec, casts, data=cell_data, offset=offset))
  File "C:\Users\drago\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pypoe-1.0.0a0-py3.9.egg\PyPoE\poe\file\dat.py", line 763, in _cast_from_spec
    value.append(self._cast_from_spec(specification, casts[1:], value, data_offset+i*casts[1:][0][1]))
  File "C:\Users\drago\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pypoe-1.0.0a0-py3.9.egg\PyPoE\poe\file\dat.py", line 713, in _cast_from_spec
    ivalue = data[0] if data else struct.unpack('<' + casts[0][2], self._file_raw[offset:offset+casts[0][1]])[0]
struct.error: unpack requires a buffer of 8 bytes

Any idea why this is happening?
@John-Paul-R
Copy link
Contributor

This error can sometimes occur when the spec does not match what is present in the actual dat file you are trying to read.

@Dragony
Copy link
Author

Dragony commented May 19, 2021

OK thanks. I've seen the Specification row size ... error a lot and wasn't sure if this one was from the same category.

@John-Paul-R
Copy link
Contributor

To be a bit more specific: I'm not sure if that is the only thing that can cause this error, but it's the only thing that I've personally had cause it.

@Dragony
Copy link
Author

Dragony commented May 20, 2021

Do you know if it is reasonable to expect an update from the maintainers? Or do you think I will have more luck fixing this myself?

@John-Paul-R
Copy link
Contributor

John-Paul-R commented May 20, 2021 •
edited

checks project's commit history

It seems that some people occasionally submit specification updates, but I don't see a standard release cycle anywhere.

Generally, if I have a spec error I fix it myself. (There are instructions for how to name different types of fields in fields.py). If you do so, consider submitting your spec update as a PR, so that everyone can benefit.

If you wait, there is a chance someone might come along and update it, but there is no official update strategy/deadline afaik.

E: I imagine many people simply update the spec for just the files that they need to reference, and leave the rest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Dragony @John-Paul-R