[feature](paimon) support read paimon ro (read-optimized) system table #59340
+132
−9
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
cambyzju
force-pushed
the
paimon-ro-dev-master
branch
from
a1fdae4 to
e3d1431
Compare
Contributor
Author
|
run buildall |
There was a problem hiding this comment.
Pull request overview
This PR adds support for reading Paimon read-optimized (RO) system tables, which allows users to query fully compacted data without overlapping key ranges, enabling the use of native readers for better performance.
Key Changes:
- Introduces
@rosyntax for accessing Paimon read-optimized system tables - Adds validation to ensure
@rois used without parameters - Implements RO table handling in PaimonExternalTable by fetching the "ro" system table variant
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| DorisParser.g4 | Modified grammar to make parentheses optional for scan parameters, enabling @ro syntax without parentheses |
| TableScanParams.java | Added RO constant, validation logic, and isRo() method to support read-optimized table parameter |
| PaimonExternalTable.java | Implemented RO table snapshot retrieval logic following the same pattern as branch handling |
| PaimonScanNode.java | Added table name (including $ro suffix) to explain output for better query visibility |
| test_paimon_ro_read.groovy | Added comprehensive test verifying RO read behavior and error handling for invalid parameter usage |
| test_paimon_ro_read.out | Expected test output showing difference between normal and RO reads |
| run10.sql | Test data setup script creating a Paimon table with compacted and delta data |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // - One is in map form, where the data is stored in `mapParams`. | ||
| // such as: @func_name('param1'='value1', 'param2'='value2', 'param3'='value3') | ||
| // - The other is in list form, where the data is stored in `listParams`. | ||
| // - Another other one is in list form, where the data is stored in `listParams`. |
There was a problem hiding this comment.
The phrase "Another other one" is grammatically incorrect. It should be either "Another one" or "The other one" to maintain grammatical clarity.
Suggested change
| // - Another other one is in list form, where the data is stored in `listParams`. | |
| // - Another one is in list form, where the data is stored in `listParams`. |
| throw new IllegalArgumentException("Invalid param type: " + paramType); | ||
| } | ||
| if (isRo() && (!mapParams.isEmpty() || !listParams.isEmpty())) { | ||
| throw new IllegalArgumentException("Please use '@ro' for Paimon read-optimized system table"); |
There was a problem hiding this comment.
The error message says "Please use '@ro' for Paimon read-optimized system table", but this validation is triggered when RO is used WITH parameters (like @ro('param'='value')). The message could be clearer by specifying that RO should be used without parameters, for example: "The '@ro' parameter for Paimon read-optimized tables should be used without arguments".
Suggested change
| throw new IllegalArgumentException("Please use '@ro' for Paimon read-optimized system table"); | |
| throw new IllegalArgumentException("The '@ro' parameter for Paimon read-optimized tables must be used without arguments"); |
TPC-H: Total hot run time: 35565 ms |
TPC-DS: Total hot run time: 180031 ms |
ClickBench: Total hot run time: 27.61 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Usage Example:
rotable, we could use the Native reader with some data latency:Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)