feat: Add fallback checks in nativeDataFusionScan for unsupported features#3322
feat: Add fallback checks in nativeDataFusionScan for unsupported features#3322andygrove wants to merge 4 commits intoapache:mainfrom
Conversation
Add checks for metadata columns, Parquet field ID reads, bucketed scans, and row index generation so that auto mode falls back to native_iceberg_compat. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
andygrove
changed the title
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3322 +/- ##
============================================
+ Coverage 56.12% 59.93% +3.80%
- Complexity 976 1462 +486
============================================
Files 119 175 +56
Lines 11743 16173 +4430
Branches 2251 2685 +434
============================================
+ Hits 6591 9693 +3102
- Misses 4012 5129 +1117
- Partials 1140 1351 +211 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
comphead
left a comment
comphead
left a comment
There was a problem hiding this comment.
Thanks @andygrove
I'm thinking we should have generate some doc from withInfo to see all the unsupported features and it would also generate a scope of future work
| } | ||
|
|
||
| if (scanExec.bucketedScan) { | ||
| withInfo(scanExec, "Native DataFusion scan does not support bucketed scans") |
There was a problem hiding this comment.
Yes it does, I think we just fail the Spark SQL test for not formatting the explain output in the same way. I'm confused how this PR is passing the test that #2888 added.
There was a problem hiding this comment.
I see. Thanks. I will move this to draft for now. Here is the related issue for the Spark SQL test failures, and yes, it looks like we need to update the diff for these. #3319
There was a problem hiding this comment.
See #3392 for Spark SQL test updates related to bucketed scans. I will remove this fallback from this PR.
Bucketed scans should work with native DataFusion scan without requiring a fallback to Spark. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
|
I have now incorporated these changes into #3393 |
4 participants
Part of #3312
Part of #3317
Part of #3319
Summary
CometNativeScan.isSupported()for metadata columns, bucketed scans, and row index generation🤖 Generated with Claude Code