Skip to content

WIP: Add live device screen streaming to the Screen Capture window - #217

Open
todi1856 wants to merge 106 commits into
masterfrom
livestream
Open

todi1856 wants to merge 106 commits into
masterfrom
livestream

Conversation

@todi1856

@todi1856 todi1856 commented Sep 15, 2026

Copy link
Copy Markdown
Member

Type of change:

  • Bug fix
  • Improvement
  • Code refactor
  • Breaking change
  • Documentation update
  • Other (please describe)

Description

The main logcat window is not affected by this change, only screen capture window is affected

Video

You can now watch the device's screen live in the Screen Capture window, and click, scroll and type into it from the Editor.

Also, it's not accessible via Window->Analysis->Android Screen Capture

Capturing needs something on the device: External/UnityLogcatServer is a small Gradle project that builds a 17 KB jar, run through app_process as the shell user. It mirrors a display into an ImageReader, JPEG-encodes each frame and writes it over an abstract socket that adb forward exposes. Frames are only produced when the screen changes, so an idle device costs almost nothing.

Input travels back up the same socket: touch, scroll wheel, keys (through the device's own keyboard layout), and Ctrl/Cmd + A, C, V for select-all/copy/paste on the device. Back / Home / Overview buttons sit beside the image, which is the only way in on gesture-navigation devices.

The window was reorganised around it. Screenshots used to overwrite one file in Temp; they're now kept in Library/AndroidLogcat/Screenshots and listed down the left, with the live view as the first row — arrow keys to cycle, F2 rename, Del delete, right-click for Show In Explorer / Open / Save As, Reconnect on the Live row, Ctrl+Shift+S to capture. The window also opens from Window > Analysis > Android Screen Capture. Stream size, JPEG quality and frame rate cap are in Preferences > Analysis > Android Logcat Settings.

Improved screenshot window

image

Settings

image

Checklist for PR maker

  • Have you added a backport label? (new feature, no backport intended — confirm)
  • Have you updated the Changelog?
  • Have you added or updated the Documentation to your PR?

Testing status

  • Existing or new automation tests - what automation was added, changed
  • New tests are covering
    • start/stop live steam
    • frame delivery
    • touch, keys
    • scroll
    • restart
    • streaming against a device that was asleep
    • screenshot list, delete, rename and external-change tests.

Devices:

  • Pixel 2 (Android 10)
  • Pixel 9 Pro Fold (Android 16)
    • Folding/Unfolding while viewing
  • CPH1831 (Android 8.1).

Testing checklist

  • Built and run editor Locally or Yamato
  • Run on device Android/iOS (if applicable)
  • All items have tooltips?

@u-pr u-pr Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Harness Review

LGTM

The live-stream implementation and packaging integration are cohesive overall, but secondary-display input handling contains a concrete reflection-cache flaw.

Reviewed commit f641860

🤖 Helpful? 👍/👎

@todi1856

Copy link
Copy Markdown
Member Author

/review

@u-pr u-pr Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Harness Review

Needs changes

The live-streaming and screenshot-history additions are substantial, but a few reachable platform and edge paths remain incorrect.

Reviewed commit 5544c4c

🤖 Helpful? 👍/👎

// belongs to the Layout Viewer as much as to this window.
else if (!m_ScreenshotList.DoPreviewGUI(imageRect))
{
var message = m_DeviceSelection.SelectedDevice == null

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1

This no-device message is unreachable because OnGUI bypasses DoPreviewGUI whenever SelectedDevice is null. Consequently, disconnecting the last device hides the entire device-independent screenshot list, preventing users from viewing, exporting, renaming, or deleting saved captures. Continue rendering the screenshot list without a device and gate only capture/live-stream actions.

🤖 Helpful? 👍/👎

var process = new System.Diagnostics.Process();
var si = process.StartInfo;
si.WorkingDirectory = projectDirectory;
si.FileName = windows ? Path.Combine(projectDirectory, "gradlew.bat") : "sh";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1

On Windows this assigns a .bat file directly to ProcessStartInfo.FileName while UseShellExecute is false. That path uses CreateProcess, which cannot execute batch files directly, so the offered “Build the server jar” action fails with a Win32Exception. Launch the wrapper through cmd.exe /c on Windows.

🤖 Helpful? 👍/👎

private IEnumerator WaitForMoreFrames(string what, int framesBefore, int count)
{
return WaitForCondition(what,
() => Runtime.LiveStream.FramesReceived > framesBefore + count,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2

This condition requires count + 1 new frames, despite callers requesting exactly count. If the device produces exactly five frames for the animation and then becomes static, the test asking for five waits until timeout. Use >= framesBefore + count.

🤖 Helpful? 👍/👎

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant