Interacting with Unreal Engine

Interacting with Unreal Engine

One of the core capabilities of the Arcware Pixel Streaming WebSDK is enabling direct communication between your web application and the Unreal Engine application running on the streaming instance.

This allows your web interface to control the Unreal experience in real time — for example by triggering gameplay actions, changing settings, switching scenes, or sending custom commands.

This interaction happens through bidirectional messaging over the WebRTC data channel.

---

Overview

The WebSDK enables two main interaction flows:

Direction	Purpose
Web → Unreal Engine	Send commands or UI events
Unreal Engine → Web	Receive responses or data from Unreal Engine

These interactions are handled through the PixelStreaming instance.

Method / Handler	Purpose
emitUIInteraction()	Send a message from the web application to Unreal Engine
applicationResponseHandler	Receive responses sent from Unreal Engine

These APIs form the foundation for building custom UI controls, dashboards, menus, or any other web-based interaction with your Unreal Engine experience.

---

Sending Messages to Unreal Engine

To send input or commands to Unreal Engine, use the emitUIInteraction() method.

This method is exposed by the PixelStreaming instance and works the same way for both:

• UI integrations (ArcwareInit)

• headless integrations (CoreSetup)

PixelStreaming.emitUIInteraction(descriptor: object | string): void

The descriptor can be either:

Type	Description
object	JSON object sent to Unreal Engine
string	Raw string message

The exact structure depends on how your Unreal Engine project handles incoming messages.

---

Basic Example

PixelStreaming.emitUIInteraction({
  action: "Jump"
});

Unreal Engine can then interpret this message and trigger the corresponding gameplay logic.

---

Example: UI Button Trigger

document.getElementById("jump-button").addEventListener("click", () => {
  PixelStreaming.emitUIInteraction({
    action: "Jump"
  });
});

Example HTML:

<button id="jump-button">
Jump
</button>

---

Example: Structured Command

Commands can contain structured payloads.

PixelStreaming.emitUIInteraction({
  type: "playerAction",
  payload: {
    action: "Teleport",
    location: {
      x: 120,
      y: 340,
      z: 50
    }
  }
});

This approach allows you to design a flexible communication protocol between your frontend and Unreal Engine.

---

Sending a Simple String Message

If preferred, messages can also be sent as plain strings.

PixelStreaming.emitUIInteraction("openMenu");

Unreal Engine can interpret this string according to the project’s logic.

---

Receiving Responses from Unreal Engine

The PixelStreaming instance exposes a response handler that allows your web application to react to messages sent from Unreal Engine.

PixelStreaming.applicationResponseHandler?: (response: string) => void

Whenever Unreal Engine sends a response through the WebRTC data channel, this handler will be invoked.

---

Example: Handling Unreal Responses

PixelStreaming.applicationResponseHandler = (response) => {
  console.log("Received response from Unreal Engine:", response);
};

---

Example: Updating the UI

Unreal Engine can send application state updates back to the web interface.

PixelStreaming.applicationResponseHandler = (response) => {
  const data = JSON.parse(response);

  if (data.type === "scoreUpdate") {
    document.getElementById("score").innerText = data.value;
  }
};

Example UI:

<div>
Score: <span id="score">0</span>
</div>

---

Example: Screenshot Workflow

PixelStreaming.emitUIInteraction({
  CreateScreenshot: "1920x1080"
});

When Unreal Engine processes this request, it can generate the screenshot and send a response message back to the browser.

The response will be received through the applicationResponseHandler.

---

How the Messaging Works

All communication is transmitted through the WebRTC data channel established when the Pixel Streaming connection is created.

Web Browser
      ↓
WebSDK (PixelStreaming)
      ↓
WebRTC Data Channel
      ↓
Unreal Engine Application

Because this channel stays active while the stream is connected, messages can be exchanged continuously with very low latency.

---

Important Considerations

The WebSDK is responsible for transmitting messages between the browser and the Unreal Engine instance.

Your Unreal Engine project must implement the logic to:

• receive incoming messages

• interpret command structures

• execute gameplay or application logic

• send responses back to the browser

Without corresponding handlers in Unreal Engine, messages sent via emitUIInteraction() will not trigger any behavior.

---

Unreal Engine Setup

Instructions on how to configure Unreal Engine to process WebSDK messages can be found here:

Pixel Streaming Input / Json messages