# Interacting with Unreal Engine

## Interacting with Unreal Engine

One of the core capabilities of the **Arcware Pixel Streaming WebSDK** is enabling **direct communication between your web application and the Unreal Engine application** running on the streaming instance.

This allows your web interface to control the Unreal experience in real time — for example by triggering gameplay actions, changing settings, switching scenes, or sending custom commands.

This interaction happens through **bidirectional messaging over the WebRTC data channel**.

***

## Overview

The WebSDK enables two main interaction flows:

| Direction           | Purpose                                      |
| ------------------- | -------------------------------------------- |
| Web → Unreal Engine | Send commands or UI events                   |
| Unreal Engine → Web | Receive responses or data from Unreal Engine |

These interactions are handled through the **PixelStreaming instance**.

| Method / Handler             | Purpose                                                  |
| ---------------------------- | -------------------------------------------------------- |
| `emitUIInteraction()`        | Send a message from the web application to Unreal Engine |
| `applicationResponseHandler` | Receive responses sent from Unreal Engine                |

These APIs form the foundation for building **custom UI controls**, dashboards, menus, or any other web-based interaction with your Unreal Engine experience.

***

## Sending Messages to Unreal Engine

To send input or commands to Unreal Engine, use the `emitUIInteraction()` method.

This method is exposed by the **PixelStreaming instance** and works the same way for both:

* **UI integrations (`ArcwareInit`)**
* **headless integrations (`CoreSetup`)**

```typescript
PixelStreaming.emitUIInteraction(descriptor: object | string): void
```

The `descriptor` can be either:

| Type   | Description                       |
| ------ | --------------------------------- |
| object | JSON object sent to Unreal Engine |
| string | Raw string message                |

The exact structure depends on how your Unreal Engine project handles incoming messages.

***

## Basic Example

```javascript
PixelStreaming.emitUIInteraction({
  action: "Jump"
});
```

Unreal Engine can then interpret this message and trigger the corresponding gameplay logic.

***

## Example: UI Button Trigger

```javascript
document.getElementById("jump-button").addEventListener("click", () => {
  PixelStreaming.emitUIInteraction({
    action: "Jump"
  });
});
```

Example HTML:

```html
<button id="jump-button">
Jump
</button>
```

***

## Example: Structured Command

Commands can contain structured payloads.

```javascript
PixelStreaming.emitUIInteraction({
  type: "playerAction",
  payload: {
    action: "Teleport",
    location: {
      x: 120,
      y: 340,
      z: 50
    }
  }
});
```

This approach allows you to design a flexible communication protocol between your frontend and Unreal Engine.

***

## Sending a Simple String Message

If preferred, messages can also be sent as plain strings.

```javascript
PixelStreaming.emitUIInteraction("openMenu");
```

Unreal Engine can interpret this string according to the project’s logic.

***

## Receiving Responses from Unreal Engine

The PixelStreaming instance exposes a response handler that allows your web application to react to messages sent from Unreal Engine.

```typescript
PixelStreaming.applicationResponseHandler?: (response: string) => void
```

Whenever Unreal Engine sends a response through the WebRTC data channel, this handler will be invoked.

***

## Example: Handling Unreal Responses

```javascript
PixelStreaming.applicationResponseHandler = (response) => {
  console.log("Received response from Unreal Engine:", response);
};
```

***

## Example: Updating the UI

Unreal Engine can send application state updates back to the web interface.

```javascript
PixelStreaming.applicationResponseHandler = (response) => {
  const data = JSON.parse(response);

  if (data.type === "scoreUpdate") {
    document.getElementById("score").innerText = data.value;
  }
};
```

Example UI:

```html
<div>
Score: <span id="score">0</span>
</div>
```

***

## Example: Screenshot Workflow

```javascript
PixelStreaming.emitUIInteraction({
  CreateScreenshot: "1920x1080"
});
```

When Unreal Engine processes this request, it can generate the screenshot and send a response message back to the browser.

The response will be received through the `applicationResponseHandler`.

***

## How the Messaging Works

All communication is transmitted through the **WebRTC data channel** established when the Pixel Streaming connection is created.

```
Web Browser
      ↓
WebSDK (PixelStreaming)
      ↓
WebRTC Data Channel
      ↓
Unreal Engine Application
```

Because this channel stays active while the stream is connected, messages can be exchanged continuously with very low latency.

***

## Important Considerations

The WebSDK is responsible for **transmitting messages** between the browser and the Unreal Engine instance.

Your Unreal Engine project must implement the logic to:

* receive incoming messages
* interpret command structures
* execute gameplay or application logic
* send responses back to the browser

Without corresponding handlers in Unreal Engine, messages sent via `emitUIInteraction()` will not trigger any behavior.

***

## Unreal Engine Setup

Instructions on how to configure Unreal Engine to process WebSDK messages can be found here:

[Pixel Streaming Input / Json messages](/unreal-engine-setup/set-up-pixel-streaming-in-your-own-project/core-settings/pixel-streaming-input-json-messages.md)


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.arcware.cloud/web-integration/new-websdk/interacting-with-unreal-engine.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.