Qualcomm AI Engine Direct - Support Debug Handle and Integrate IntermediateOutputCapturer #16316
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/16316
Note: Links to docs will display an error until the docs builds have been completed. ❌ 1 New Failure, 1 Unrelated FailureAs of commit 00c4f7e with merge base 3233761 ( NEW FAILURE - The following job has failed:
UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
bot
added
the
CLA Signed
This PR needs a
|
|
Is the PR ready to be reviewed now? |
| For any passes executed during qnn_preprocess, users will need to handle debug_handle ID themselves. | ||
| Description: During passes transformation, some passes might be copying some node's meta when creating a new node, | ||
| which means multiple nodes might be sharing the same debug_handle ID while it shouldn't. |
There was a problem hiding this comment.
im not super understand here: if several nodes comes from one acient node (e..g doing decomposition on some op), they should have the same debug handle for tracing.
There was a problem hiding this comment.
I think the idea is that if we decompose the node but never assign a new handle ID, we are just saving the information for the last decomposed node rather than all decomposed node. I have draw an example below. Since edge and QNN has 1 to 1 mapping in this case, I think it would be better to gather all possible information rather than the last node's debug info. Since we reassign graph_handle, instead of only getting the output of node2, we can also get info for node1.
There was a problem hiding this comment.
here im a little confused: when we see the qnn graph, how can we know that the qnn_node_1 and qnn_node2 here comes from a same super node? Or another q might be, which graph will play as the ground truth graph, when you doing intermediate comparsion?
gather all possible information rather than the last node's debug info.
We won't gather only the last node debug info, but all info.
In ExecuTorch normally we follow this rule:
if we transform {old_node_1, old_node_2, ..., old_node_n} into {new_node_1, new_node_2, ..., new_node_m}, where n and m can be arbitrary number starting from 1, then: eery new_node should have same debug handle, and the debug handle will be set(old_node_1.debug_handle + old_node_2.debug_handle, ..., old_node_n.debug_handle)
you can see if n is 1, this transform will be a operator decomposition; if m is 1, this transform will be a operator fusion, etc.
In this way whenever we see an arbitrary new_node, we will know its ancestor.
Not sure if that make sense to you?
winskuo-quic
force-pushed
the
dev1/winskuo/debug_handle
branch
2 times, most recently
from
9a7ca59 to
dc72614
Compare
Hi @Gasoonjia, |
|
Hi @cccclai, @Gasoonjia, @kimishpatel, Also, I would also like to get some suggestions on the official API to retrieve an edge IR. The current way of retrieving an edge IR is through: executorch/examples/qualcomm/utils.py Line 499 in 0fb422f However, I encountered following issues when retrieving edge IR using the above method.
Thanks |
|
I think instead of using the edge graph IR as the ground truth for comparsion, it will be great if we can use the export program ET stack get at the first place (e.g. the export graph of executorch/examples/qualcomm/utils.py Line 480 in 0fb422f You can see how we calculate intermediate output numercal descrepancy: executorch/devtools/inspector/_inspector.py Line 1407 in 0fb422f https://github.com/pytorch/executorch/blob/0fb422f9c59e0e5526c0082352a583baf0510fb7/exir/passes/debug_handle_generator_pass.py here's pass for debug handle generation, where the debug handle of a node is the same as the node sharing the same greatest ancestor node in the export flow. |
|
Here's an example of how our current API works on VIT model on xnnpack backend: https://gist.github.com/Gasoonjia/db6285ac39ad5759b95c7a92d37cd4f8 and below is the expected output. For some ops like layernorm there're still some issue i need to fix.
I would love to chat with you regarding how we can make the pipeline works on qualcomm backend! |
…ediateOutputCapturer
winskuo-quic
force-pushed
the
dev1/winskuo/debug_handle
branch
from
dc72614 to
00c4f7e
Compare
|
Hi @Gasoonjia, |
| import operator | ||
|
|
||
| import torch | ||
| from executorch.backends.qualcomm.utils.constants import QCOM_DEBUG_HANDLE |
There was a problem hiding this comment.
I would love to directly reuse DEBUG_HANDLE_KEY from ExecuTorch https://github.com/pytorch/executorch/blob/main/exir/passes/debug_handle_generator_pass.py#L10 to make sure that we are working on same item.
| def call(self, graph_module: torch.fx.GraphModule): | ||
| handle_counter = 1 | ||
| visited = set() | ||
| for node in graph_module.graph.nodes: |
There was a problem hiding this comment.
not sure if Qualcomm can handle conditional graph. If so i think the way you are adding debug handle might not able to equip debug handle to all branches. You can follow what i'm doing here:
https://github.com/pytorch/executorch/blob/main/exir/passes/debug_handle_generator_pass.py#L14
| assert ( | ||
| source_node.name in visited | ||
| ), "Graph is not traversed in topological order, unexpected behavior." | ||
| node.meta[QCOM_DEBUG_HANDLE] = source_node.meta[QCOM_DEBUG_HANDLE] |
There was a problem hiding this comment.
Curious why do we need to set the get_item node the same debug handle as the soruce node? Since it will introduce duplicate debug handle in the graph and im a little bit worried if it could cause any issue in the down stream,
| tensor_name = f"{node.name}_{wrapper_idx}" | ||
|
|
||
| # Only append special namings when enable tensor dump, since longer name results bigger .pte | ||
| if (handle_id := node.meta.get(QCOM_DEBUG_HANDLE)) and self.enable_tensor_dump: |
There was a problem hiding this comment.
wondering if we still need this file since we will migrate to devtool infra?
| This class serves as an intermediate point and is inserted right after the call_function node. | ||
| It also saves some metadata such as scale, offset, etc. | ||
| Since we just want to check the intermediate output, we will directly return the value during the forward call. | ||
| class QNNIntermediateDebugger: |
There was a problem hiding this comment.
I think we can change or update the class target; from the comments it plays as the same role of Inspector.calculate_nunmeric_gap()
2 participants
Summary
Additional Topics:
Test plan
python backends/qualcomm/tests/test_qnn_delegate.py -k TestExampleUtilsScript.test_intermediate_debugger -s $DEVICE --model SM8650 --build_folder build-android/ --executorch_root . --image_dataset ../imagenet-mini/val/ --artifact ./e2e_test_debugpython backends/qualcomm/tests/test_qnn_delegate.py -k TestQNNQuantizedUtils.test_qnn_backend_dump_intermediate_outputs_simple_model --model SM8550 --device $DEVICE --build_folder build-androidpython backends/qualcomm/tests/test_qnn_delegate.py -k TestQNNQuantizedUtils.test_qnn_backend_dump_intermediate_outputs_topk --model SM8550 --device $DEVICE --build_folder build-android