RDKDEV-1148: Release CDM resources when not used

[marcin-mielczarczyk-red]

When HTMLMediaElement has been created in WPE and HTMLMediaElement::setMediaKeys() method was called to set DRM keys, CDMThunder implementation already allocates CDM resources and they're not released until HTMLMediaElement object is destructed.
HTMLMediaElement destruction is non-deterministic and it's relying on Garbage Collector.
In this case very often we observe that after WPE exit (WPE process is not closed, but some #boot address is being loaded), CDM resources are not released immediately and CDM can't be used by other applications like IP Player or Netflix.

Proposed solution introduces following changes:

1. CDMPrivateThunder implementation doesn't occupy CDM resources permanently, as it only needs CDM resources to check if server certificate is supported so CDM resources can be acquired before check and released afterwards

2. CDMInstanceThunder object's CDM resources are released on HTMLMediaElement::stop() to not wait for Garbage Collector
   2c36c43

Build-Tests	Layout-Tests
✅ 🛠 wpe-238-amd64-build	✅ 🧪 wpe-238-amd64-layout
✅ 🛠 wpe-238-arm32-build	✅ 🧪 wpe-238-arm32-layout

[calvaris]

I would need a test case, please.

[marcin-mielczarczyk-red]

@calvaris is manual test case enough (description how to reproduce the problem)?

[calvaris]

@calvaris is manual test case enough (description how to reproduce the problem)?

yes.

[calvaris]

but it has to show that the resources are not released before the patch and are released after building with the patch.

[marcin-mielczarczyk-red]

To reproduce the problem below steps should be performed:

1. start application which uses DRM (most often it reproduces in Apple or Film1)
2. start video playback and keep playing for 2-3 minutes
3. exit application (or load URL from a different domain, for example "google.com")
4. repeat above steps multiple times
   Sometimes (once per 20 times or even more seldom) DRM session is not released because HTMLMediaElement is not freed (seems that reference is kept by some object). Triggering garbage collector manually doesn't help (I tried multiple times).
   Without proposed patch the only way to release DRM session is to load "about:blank", which causes freeing of HTMLMediaElement and frees DRM session in result.

[eocanha]

Might the retained HTMLMediaElement issue have been solved by this recent commit (in wpe-2.46, might be applicable to wpe-2.38 too)?
4c72cef

[marcin-mielczarczyk-red]

Might the retained HTMLMediaElement issue have been solved by this recent commit (in wpe-2.46, might be applicable to wpe-2.38 too)? 4c72cef

This patch is applicable to 2.38 and looks related to issue we're observing.
Original root cause of DRM not being released is a leaked HTMLMediaElement so that would be a better solution.
Thank you for pointing it out. Will perform tests and update this ticket.

[eocanha]

Patch proactively backported it to wpe-2.38 as a087bcd

[marcin-mielczarczyk-red]

Unfortunately HTMMediaElement patch didn't solve issue with DRM release which I reported here.
I'm still able to reproduce the issue with proposed patch applied.
In such case could you please review my solution?

[eocanha]

I've added this task to my to-do list and I'll be taking it after I finish other tasks with more priority.

Would it be possible to have a minimal automated (not manual) test case, even if it's by "simulating" the <video> element leak by adding it to the page, using it, removing it from he page but still hold a reference to it (assuming that this behaviour calls ActiveDOMObject::stop(), implemented by HTMLVideoElement::stop())? At least in that way we could ensure that the PR actually releases the CDM resources.

[marcin-mielczarczyk-red]

@eocanha I'm not sure how to implement such test.
Would you like it to be a part of automated tests located in "./Tools/TestWebKitAPI/Tests/WebKit/" directory?

[eocanha]

The usual way we test this kind of things upstream is by creating tests under the LayoutTest directory (eg: platform/wpe/media/encrypted-media/). However, layout tests can't be run on the devices (only upstream, where ThunderCDM isn't available). For this reason, I think it should be enough just to supply a simple standalone html webpage that would try to load an encrypted content and then do something else and prove that the HTMLMediaElement is still around, taking the CDM resources.

The idea is not to have to depend on commercial video services to reproduce the issue, but having a simple standalone reproducing test case. I can later work on adapting that test case to become a layout test if that makes sense.

[marcin-mielczarczyk-red]

Clear. Thank you for the update. I'll try to prepare such test webpage.
BTW: are you able to run wpe minibrowser (cog) on x86-64 host like it's described on below webpage?
https://wpewebkit.org/blog/04-wpe-qa-tooling.html

I was able to build WPE using flatpak, ran tests successfully but running minibrowser just hangs (I'm using Wayland environment, no issues with permission, just hang without any errors). I'm wondering if anyone is using minibrowser on PC, or it's not maintained.

[eocanha]

Hmmm... no, not cog. I run WPE through WPEFramework on a Raspberry Pi 3 using https://github.com/WebPlatformForEmbedded/buildroot/tree/wpe with the https://github.com/WebPlatformForEmbedded/buildroot/blob/wpe/configs/raspberrypi3_wpe_ml_defconfig configuration, or on a Sagem box using https://github.com/WebPlatformForEmbedded/buildroot/blob/wpe/configs/sagemcomrdk_wpe_ml_defconfig. Both require access (provided by Comcast) to some restricted repositories to enable Playready and Widevine. I don't normally use cog.
I only develop on WPE/WebKitGTK on x86_64 desktop for upstream main (not 2.38 nor 2.46), and in any case, we don't have any way to make ThunderCDM work there. The development environment that we use there is based on https://github.com/Igalia/webkit-container-sdk instead of on the now deprecated Flatpak SDK.

[marcin-mielczarczyk-red]

@eocanha I tried to prepare some test page which plays DRM content and leaks HTMLMediaElement, but so far I was not successful.
You can find a test page in the attachment, which plays PlayeReady content for 30s and than stops and keeps the reference.
drm_test.html

You can set this URL multiple times and you'll observe that reference count for DRM resources increase but at the end they are always released by garbage collector (I was testing it for couple of hours).
I assume that vendor apps have some corner case we're not aware of.
You should be able to notice that with proposed patch DRM is released immediately on a playback stop and we don't have to wait for GC to kick in, which allows other applications to use DRM immediately after browser exit.
Even if we would fix HTMLMediaElement leak, relying on GC to release DRM resources is not good solution for us as some applications need to start playback immediately after browser exit (i.e. ipplayer app which is showing linear channels).

[eocanha]

Even if the patch looks good on first sight, we're worried about potential problems when the MediaKeys object attached to the <video> is reused with a new and completely different <video> (yes, it's a strange use case, but it must be supported).

To ensure that, I've checked the test case here and even enhanced it with extra checks, to:

• Reuse the dash.js player, the video tag and the MediaKeys
• Reset player, reuse video, don't reuse MediaKeys
• Reset player, reuse video, reuse MediaKeys
• Reset player, don't reuse video, reuse MediaKeys

Encrypted playback worked in all the cases. However, some of the scenarios where I thought that the old MediaKeys object was being used revealed that a new one was actually being used instead (probably because the dash.js player held and used the new instance and, when I set the old one manually, the new one kept being used instead). I checked this by printing the CDMInstanceThunder pointer to the logs from C++. CDMInstanceThunder is a subclass of CDMInstance, which is an object held by MediaKeys, so if the instance is the same, it means that the MediaKeys is also the same (since the latter "owns" the former).

Having this into account, I had to resort to a more "manual" kind of test for decryption. I used eme-v3-clean.html as base and modified it to reuse MediaKeys, ending up with eme-v3-reuse-mediakeys.html. When that happened, an assert in open_cdm_ext.cpp was triggered.

I still need to analyze a bit more what's the meaning of that assert. I'll report my findings here.

[eocanha]

I've added some extra code in branch eocanha/eocanha-162 to reinstantiate the OpenCDMSystem when a new instance is requested, avoiding the crash that I detected some days ago.

However, after a careful discussion with @calvaris (who can comment more on the topic), we think that this solution, even if seems to work, is messing up with the lifecycles of objects and can create unforeseen problems in the future. This solution will never be accepted upstream, and we're not merging it in wpe-2.38 either because we don't want to risk having the problems that it may create. If LibertyGlobal decides to integrate it on their own repository, it would be at their own risk.

Since the EME spec doesn't support an "invalid state" for MediaKeys (where it has internally released all its resourced and isn't usable anymore), the underlying objects should be kept alive until MediaKeys and the other objects depending from it are garbage collected and destroyed. The proper solution would involve a coordinated effort between Thunder and WebKit to ensure that when the WebKitBrowser plugin is destroyed (disabled), or even suspended, the webpages and their associated JavaScript objects are fully garbage collected.

[calvaris]

Or also as simple as destroying the webview or the webprocess, which is what happens when you switch to a different website if I am not mistaken. The resources are forced to be freed in that case, but as eocanha said, landing this PR is a bad idea as you would have JS objects with dead platform counterparts, leading to dragons.