[SPARK-54847][BUILD] unify the proto/antlr output folder between sbt and maven #53616
Conversation
cloud-fan
changed the title
pan3793
left a comment
pan3793
left a comment
There was a problem hiding this comment.
thanks, this does fix the issue about maven/sbt mix-used cases!
the only left noisy is my IDEA still seems not to correctly recognize classes overridden in spark-connect-shims, thus can not jump code from IDEA
.sbtopts
Outdated
| @@ -0,0 +1,3 @@ | |||
| -J-Xmx4g | |||
| -J-Xms4g | |||
There was a problem hiding this comment.
to make the build more stable, avoid OOM
There was a problem hiding this comment.
Pull request overview
This PR unifies the output directories for generated sources (protobuf and ANTLR4) between sbt and Maven builds to prevent duplicate class issues when developers use both build systems (e.g., local testing with sbt and IDE with Maven).
- Updates sbt build configuration to use Maven's default output directories for generated sources
- Adds
.sbtoptsfile to configure JVM memory settings for sbt
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| project/SparkBuild.scala | Updates output directories for protobuf (Core, SQL, SparkConnectCommon modules) and ANTLR4 (SqlApi module) generated sources to match Maven's default paths |
| .sbtopts | Adds new sbt configuration file with JVM memory settings (4GB Xmx/Xms, 1GB MaxMetaspaceSize) |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
the last commit simply updates |
cloud-fan
changed the title
…and maven ### What changes were proposed in this pull request? People may use sbt and maven together: local test with sbt, IDE with Maven. As of today this will trigger duplicated class issue during compilation, because sbt and Maven use different output folder for generated proto/antlr java classes. This PR updates the sbt build to use the same folder as Maven for proto/antlr output. ### Why are the changes needed? Improve dev experience. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manual test ### Was this patch authored or co-authored using generative AI tooling? cursor 2.2.43 Closes #53616 from cloud-fan/shim. Lead-authored-by: Wenchen Fan <[email protected]> Co-authored-by: Wenchen Fan <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit 174b62d) Signed-off-by: Wenchen Fan <[email protected]>
3 participants
What changes were proposed in this pull request?
People may use sbt and maven together: local test with sbt, IDE with Maven. As of today this will trigger duplicated class issue during compilation, because sbt and Maven use different output folder for generated proto/antlr java classes. This PR updates the sbt build to use the same folder as Maven for proto/antlr output.
Why are the changes needed?
Improve dev experience.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Manual test
Was this patch authored or co-authored using generative AI tooling?
cursor 2.2.43