Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[E2B-457] Unable to create custom templates with Dockerfiles containing more than ~4.3GB of local data #278

Closed
mlejva opened this issue Dec 31, 2023 · 2 comments
Closed

[E2B-457] Unable to create custom templates with Dockerfiles containing more than ~4.3GB of local data #278

mlejva opened this issue Dec 31, 2023 · 2 comments
Labels
bug Something isn't working

Comments

@mlejva
Copy link
Member

mlejva commented Dec 31, 2023 •
edited

When a user is creating a custom template, they specify a Dockerfile. If the Dockerfile requires to copy more than 4,294,967,295 bytes of a local data, it will fail with the following message

Preparing sandbox template building (35639 files in Docker build context).
Found ./e2b.Dockerfile that will be used to build the sandbox template.
node:internal/blob:165
      throw new ERR_BUFFER_TOO_LARGE(0xFFFFFFFF);
      ^
RangeError [ERR_BUFFER_TOO_LARGE]: Cannot create a Buffer larger than 4294967295 bytes
    at new NodeError (node:internal/errors:399:5)
    at new Blob (node:internal/blob:165:13)
    at R1.<anonymous> (/Users/default/Library/pnpm/global/5/.pnpm/@e2b+cli@0.1.20_openai@4.24.1/node_modules/@e2b/cli/dist/index.js:104:1093)
    at R1.emit (node:events:513:28)
    at n1.updateNonPrimary (/Users/default/Library/pnpm/global/5/.pnpm/@e2b+cli@0.1.20_openai@4.24.1/node_modules/@e2b/cli/dist/index.js:54:11372)
    at n1.update (/Users/default/Library/pnpm/global/5/.pnpm/@e2b+cli@0.1.20_openai@4.24.1/node_modules/@e2b/cli/dist/index.js:54:11193)
    at n1.ez (/Users/default/Library/pnpm/global/5/.pnpm/@e2b+cli@0.1.20_openai@4.24.1/node_modules/@e2b/cli/dist/index.js:54:13685)
    at process.processTicksAndRejections (node:internal/process/task_queues:77:11) {
  code: 'ERR_BUFFER_TOO_LARGE'
}
Node.js v18.16.1

We're currently using Node's Blob to represent the files and upload them all at once. One of the potential solutions to this issue is to start streaming the data in chunks to the backend.

From SyncLinear.com | E2B-457
@mlejva mlejva changed the title Unable to create custom templates with Dockerfiles containing more than ~4.3GB of local data [E2B-457] Unable to create custom templates with Dockerfiles containing more than ~4.3GB of local data Dec 31, 2023
@mlejva
Copy link
Member Author

mlejva commented Dec 31, 2023 •
edited

The maximum size of a Buffer in Node is described here.

On 32-bit architectures, this value currently is 230 - 1 (about 1 GiB).
On 64-bit architectures, this value currently is 232 (about 4 GiB).

@ValentaTomas ValentaTomas added the bug Something isn't working label Jan 8, 2024
@jakubno
Copy link
Member

jakubno commented Jan 23, 2024

Solved by fc037a2

@jakubno jakubno closed this as completed Jan 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mlejva @ValentaTomas @jakubno