Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Langchain integration #133

Closed
matthoffner opened this issue Jun 22, 2023 · 4 comments
Closed

Langchain integration #133

matthoffner opened this issue Jun 22, 2023 · 4 comments

Comments

@matthoffner
Copy link
Contributor

Adding web-llm as a LLM in Langchain would be really useful.

https://js.langchain.com/docs/getting-started/guide-llm

For example it would be nice to have a web-llm option that works similar to the OpenAI one.

https://github.com/hwchase17/langchainjs/tree/main/langchain/src/llms

@craigmassie
Copy link

craigmassie commented Jun 26, 2023

Hey @matthoffner, apologies for the messy code, but I just threw this together for a project I'm working on. I'll make a cleaner PR to langchain if I find the time:

import { LLM, BaseLLMParams, BaseLLMCallOptions } from "./base.js";

// @ts-ignore -- optional interface, will gracefully degrade to `any` if `@mlc-ai/web-llm` isn't installed
import type { ChatInterface } from "@mlc-ai/web-llm";

interface InitialProgressCallback {
  report: string;
}

interface WebLLMInput extends BaseLLMParams {
  model?: string;
  initCallback: (callback: InitialProgressCallback) => void;
}

interface WebLLMWorkerInput extends WebLLMInput {
  worker: Worker;
}

interface WebLLMCallOptions extends BaseLLMCallOptions {
  progressCallback: (step: any, msg: any) => void;
}

export class WebLLM extends LLM implements WebLLMInput {
  model = "vicuna-v1-7b-q4f32_0";
  initCallback: (callback: InitialProgressCallback) => void;
  worker: Worker | undefined;

  private chat: ChatInterface;

  declare CallOptions: WebLLMCallOptions;

  private constructor(fields: WebLLMInput) {
    super(fields);
    this.model = fields?.model ?? this.model;
  }

  public static async createInstance(fields: WebLLMInput) {
    const instance = new WebLLM(fields);
    const webllm = await WebLLM.imports();

    instance.chat = new webllm.ChatModule();
    instance.chat.setInitProgressCallback(fields.initCallback);

    try {
      await instance.chat.reload(instance.model);
    } catch (err: any) {
      console.log(err.stack);
      return;
    }

    return instance;
  }

  public static async createWorkerInstance(fields: WebLLMWorkerInput) {
    const instance = new WebLLM(fields);
    instance.worker = fields.worker;

    const webllm = await WebLLM.imports();
    instance.chat = new webllm.ChatWorkerClient(fields.worker);
    instance.chat.setInitProgressCallback(fields.initCallback);

    try {
      await instance.chat.reload(instance.model);
    } catch (err: any) {
      console.log(err.stack);
      return;
    }

    return instance;
  }

  /** Get the type of LLM. */
  _llmType(): string {
    return "web_llm";
  }

  async _call(
    prompt: string,
    options: this["ParsedCallOptions"]
  ): Promise<string> {
    let output: string;
    try {
      output = await this.chat.generate(prompt, options.progressCallback);
    } catch (err: any) {
      console.log(err.stack);
      throw err;
    }
    return output;
  }

  /** @ignore */
  static async imports(): Promise<typeof import("@mlc-ai/web-llm")> {
    try {
      const webllm = await import("@mlc-ai/web-llm");
      return webllm;
    } catch (e) {
      throw new Error(
        "Please install web-llm as a dependency with yarn, e.g. `yarn add @mlc-ai/web-llm`"
      );
    }
  }
}

@matthoffner
Copy link
Contributor Author

Awesome @craigmassie it could probably go into langchain eventually too.

@craigmassie
Copy link

craigmassie commented Jun 26, 2023

@matthoffner slip of the tongue, I meant a PR to langchain :)

@tqchen tqchen closed this as completed May 31, 2024
@tqchen
Copy link
Contributor

tqchen commented May 31, 2024

langchain js now have webllm integration

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants