New demo: cat detector #465
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This will detect if a cat is on the screen. By which I mean displayed on the screen, not sitting on your laptop. This is meant as a simple demo of using MSS for AI. It works as-is, but needs to be documented, and there's some bits that could do with cleanup. There are a lot of additional features that could be added, such as showing a window with bounding boxes, but that's probably more complexity than is called for here.
Owner
|
I like it, great inspiration! |
Contributor
halldorfannar
left a comment
halldorfannar
left a comment
There was a problem hiding this comment.
Excellent PR. Well documented, enjoyable to read. Just some minor improvements suggested.
| # identify what it's seeing on its cameras. | ||
| # | ||
| # For this demo, we want to tell if a cat is anywhere on the screen, not if the whole screen is a picture of a cat. | ||
| # That means that we want to use an detector, not a classifier. |
Contributor
There was a problem hiding this comment.
Suggested change
| # That means that we want to use an detector, not a classifier. | |
| # That means that we want to use a detector, not a classifier. |
| # Performance | ||
| # =========== | ||
| # | ||
| # The biggest determinant of performance is whether the model runs on a GPU or on the CPU. GPUs are extremely |
Contributor
There was a problem hiding this comment.
Maybe we should mention here, right away, that this particular model will work on both? I know this becomes clearer in the end of this section, when GPU vs CPU performance comparisons are discussed.
| # =========== | ||
| # | ||
| # The first time you run this demo, Torchvision will download a 167 MByte DNN. This is cached in | ||
| # ~/.cache/torch/hub/checkpoints on Unix. I'm not sure where it's cached on other platforms, but it will tell you. |
Contributor
There was a problem hiding this comment.
Third person was used earlier in the text. I would avoid switching to first person here. Perhaps just go with this:
Suggested change
| # ~/.cache/torch/hub/checkpoints on Unix. I'm not sure where it's cached on other platforms, but it will tell you. | |
| # ~/.cache/torch/hub/checkpoints on Unix. If you want to know where the cache is stored on other platforms, this information will be displayed after downloading the DNN. |
| import torchvision.models.detection | ||
| import torchvision.transforms.v2 | ||
|
|
||
| # You'll also need to "pip install mss pillow". |
Contributor
There was a problem hiding this comment.
Rather than assume the user is leveraging pip (I'm a fan of uv) I would suggest the more general:
Suggested change
| # You'll also need to "pip install mss pillow". | |
| # You'll also need to install mss and pillow. |
This also aligns with the earlier text where pip is suggested but it is left open for the user how to do this specifically.
| # If an image is too small, then it's got a pretty decent chance of being a false positive: it's hard to tell if a | ||
| # Discord or Slack reaction icon is a cat or something different. We ignore any results that are too small to be | ||
| # reliable. Here, this cutoff is 0.1% of the whole monitor (about 1.5 cm square on a 27" monitor, the diameter of a | ||
| # AA battery). Like the score threshold, this is just something you try and see what the model seems to be able to |
Contributor
There was a problem hiding this comment.
Suggested change
| # AA battery). Like the score threshold, this is just something you try and see what the model seems to be able to | |
| # AA battery). Like the score threshold, this is just something you try and see what the model is able to |
| preprocess = weights.transforms() | ||
|
|
||
| # The labels ("what type of object is this") that the model gives us are just integers; for this model, they're | ||
| # from 0 to 90. The English words describing them ("cat") are in a list, stored in the weight's metadata. |
Contributor
There was a problem hiding this comment.
Suggested change
| # from 0 to 90. The English words describing them ("cat") are in a list, stored in the weight's metadata. | |
| # from 0 to 90. The English words describing them (like "cat") are in a list, stored in the weight's metadata. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This will detect if a cat is on the screen. By which I mean displayed on the screen, not sitting on your laptop.
This is meant as a simple demo of using MSS for AI. It works as-is, but needs to be documented, and there's some bits that could do with cleanup.
There are a lot of additional features that could be added, such as showing a window with bounding boxes, but that's probably more complexity than is called for here.
Changes proposed in this PR
Fixes #
(...)
./check.shpassed