feat(QTDI-1914): Sample dynamic schema connector. #1114

ypiel-talend · 2025-10-22T22:33:14Z

Requirements

https://qlik-dev.atlassian.net/browse/QTDI-1914

Any code change adding any logic MUST be tested through a unit test executed with the default build
Any API addition MUST be done with a documentation update if relevant

Why this PR is needed?

What does this PR adds (design/code thoughts)?

…ement, better clazz location.

…l loaded... to continue...

…:Talend/component-runtime into ypiel/QTDI-1914_sample_connector_dyndeps

# sample-parent/sample-features/dynamic-dependencies/dynamic-dependencies-common/src/main/java/org/talend/sdk/component/sample/feature/dynamicdependencies/config/Dependency.java # sample-parent/sample-features/dynamic-dependencies/dynamic-dependencies-common/src/main/java/org/talend/sdk/component/sample/feature/dynamicdependencies/service/AbstractDynamicDependenciesService.java

# Conflicts: # sample-parent/sample-features/pom.xml

…st log + add an error messae in returned strings.

…SPI is not loaded.

…Don't fail if SPI can't be loaded.

…i in not found.

…to transmit its dynamic dependencies.

…hSPI connector output records.

…cy-common module.

Copilot

Pull request overview

This PR adds a comprehensive sample connector module demonstrating dynamic dependency loading capabilities. The implementation showcases how to use the @DynamicDependencies annotation with various configuration types (datastore, dataset, and custom configuration) and validates SPI loading from different dependency scopes.

Changes:

Added new dynamic-dependencies module with multiple sample connectors demonstrating @DynamicDependencies feature
Introduced DynamicDependenciesConfiguration annotation for marking configuration types that compute dynamic dependencies
Created supporting modules for SPI testing and classloader validation

Reviewed changes

Copilot reviewed 99 out of 110 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
sample-parent/sample-features/pom.xml	Added dynamic-dependencies module to the build
component-api/src/main/java/.../DynamicDependenciesConfiguration.java	New annotation for marking dynamic dependency configuration types
sample-parent/sample-features/dynamic-dependencies/pom.xml	Parent POM defining the dynamic-dependencies module structure
sample-parent/sample-features/dynamic-dependencies/README.md	Comprehensive documentation explaining the dynamic dependencies feature and test scenarios
sample-parent/sample-features/dynamic-dependencies/*/pom.xml	Module-specific POM files for various sample connectors
sample-parent/sample-features/dynamic-dependencies//src/main/java//.java	Implementation classes for sample connectors, services, and configuration
sample-parent/sample-features/dynamic-dependencies//src/test/java//.java	Test classes validating dynamic dependency loading

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...nt/sample/feature/dynamicdependencies/withspi/input/DynamicDependenciesWithSPIInputTest.java

.../main/java/org/talend/sdk/component/sample/feature/dynamicdependencies/config/Connector.java

...t/sample/feature/dynamicdependencies/withDynamicDependenciesConfiguration/config/Config.java

...dk/component/sample/feature/dynamicdependencies/withDataprepRunAnnotation/config/Config.java

sample-parent/sample-features/dynamic-dependencies/README.md

sonar-eks · 2026-02-02T17:29:08Z

Quality Gate failed

3.15% Duplicated Lines (%) on New Code (is greater than 3.00%)
72 New Issues (is greater than 0)

Project ID: org.talend.sdk.component:component-runtime

View in SonarQube

…nlysis.

Copilot

Pull request overview

Copilot reviewed 99 out of 110 changed files in this pull request and generated 11 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-09T17:31:02Z

...ponent/sample/feature/loadinganalysis/withspi/service/DynamicDependenciesWithSPIService.java

+    public List<String> getDynamicDependencies(@Option("theDataset") final Dataset dataset) {
+        String dep = "org.talend.sdk.samplefeature.dynamicdependencies:service-provider-from-dynamic-dependency:"
+                + loadVersion();
+        List<String> strings = Collections.singletonList(dep);
+        log.info("Dynamic dependencies with SPI: {}", strings.stream().collect(Collectors.joining(";")));
+        return strings;
+    }


The returned Maven coordinate uses a groupId (org.talend.sdk.samplefeature.dynamicdependencies) that doesn’t match the groupId declared by the modules in this PR (org.talend.sdk.component.loading-analysis). This will make the runtime/designtime resolver look for a non-existent artifact. Update the coordinate to match the actual groupId/artifactId of service-provider-from-dynamic-dependency.

Copilot · 2026-02-09T17:31:03Z

...alend/sdk/component/sample/feature/loadinganalysis/withspi/service/CustomizeClassLoader.java

+        return Stream.of(
+                // Implementation should come from a dynamic dependency
+                "org.talend.sdk.component.sample.feature.dynamicdependencies.classloadertestlibrary.serviceInterfaces.StringProviderSPIAsDependency",
+                // Implementation should come from runtime
+                "org.talend.sdk.component.sample.feature.dynamicdependencies.classloadertestlibrary.serviceInterfaces.StringProviderSPIAsDynamicDependency");


The returned class names reference ...sample.feature.dynamicdependencies... but the actual packages added in this PR are ...sample.feature.loadinganalysis.... With the current values, the customizer won’t match the intended classes/packages, so the classloader customization won’t take effect. Replace these strings with the correct org.talend.sdk.component.sample.feature.loadinganalysis... FQNs.

Copilot · 2026-02-09T17:31:03Z

...alend/sdk/component/sample/feature/loadinganalysis/withspi/service/CustomizeClassLoader.java

+        }
+
+        log.info(
+                "org.talend.sdk.component.sample.feature.dynamicdependencies.withspi.service.CustomizeClassLoader is enable,\n"


Correct grammar in the log message: “is enable” → “is enabled”.

Suggested change

"org.talend.sdk.component.sample.feature.dynamicdependencies.withspi.service.CustomizeClassLoader is enable,\n"

"org.talend.sdk.component.sample.feature.dynamicdependencies.withspi.service.CustomizeClassLoader is enabled,\n"

Copilot · 2026-02-09T17:31:03Z

...ponent/sample/feature/loadinganalysis/withspi/service/DynamicDependenciesWithSPIService.java

+    private static String loadVersion() {
+        if (version == null) {
+            try (InputStream is = DynamicDependenciesWithSPIService.class.getClassLoader()
+                    .getResourceAsStream("version.properties")) {
+                Properties props = new Properties();
+                props.load(is);
+                version = props.getProperty("version");
+            } catch (IOException e) {
+                throw new ComponentException("Unable to load project version", e);
+            }
+        }
+        return version;
+    }


getResourceAsStream("version.properties") can return null (missing resource or wrong filtering), and props.load(is) will throw a NullPointerException which is not caught here. Throw a ComponentException with a clear message when is == null, and consider also handling missing "version" property.

Copilot · 2026-02-09T17:31:04Z

...ponent/sample/feature/loadinganalysis/withspi/service/DynamicDependenciesWithSPIService.java

+
+            Properties prop = new Properties();
+            prop.load(resourceStreamFromDependency);
+            return prop.getProperty(property);


prop.getProperty(property) can return null when the property key is missing, and later JsonObjectBuilder.add(key, value) will throw if value is null. Return a non-null fallback (similar to the “resource not found” branch) or throw a ComponentException when the property is absent.

Suggested change

return prop.getProperty(property);

final String value = prop.getProperty(property);

if (value == null) {

return "The property '%s' has not been found in resource '%s', it can't retrieve its value."

.formatted(property, resource);

}

return value;

Copilot · 2026-02-09T17:31:04Z

.../sample/feature/loadinganalysis/classloadertestlibrary/spiConsumers/AbstractSPIConsumer.java

+
+    private final Optional<S> spiImpl;
+
+    protected AbstractSPIConsumer(final Class clazz) {


This uses a raw Class type, losing type safety (and it will generate warnings). Change the signature to accept a parameterized type (for example, Class<S>) and pass that through to ServiceLoader.load(...) so the consumer can be used safely without unchecked operations.

Suggested change

protected AbstractSPIConsumer(final Class clazz) {

protected AbstractSPIConsumer(final Class<S> clazz) {

Copilot · 2026-02-09T17:31:04Z

...sdk/component/sample/feature/loadinganalysis/service/AbstractDynamicDependenciesService.java

+
+    public static final String ENTRY_RUNTIME_CLASSPATH = "runtime_classpath";
+
+    public static final String ENTRY_WORKING_DIRECTORY = "Working_directory";


This entry name is inconsistent with the rest of the schema keys (ENTRY_RUNTIME_CLASSPATH, ENTRY_ROOT_REPOSITORY, etc.) which are lower snake-case. Consider changing it to working_directory (and keep README/tests aligned) to avoid surprising API consumers and reduce schema/key mismatches.

Suggested change

public static final String ENTRY_WORKING_DIRECTORY = "Working_directory";

public static final String ENTRY_WORKING_DIRECTORY = "working_directory";

Copilot · 2026-02-09T17:31:05Z

...ciesConfiguration/service/DynamicDependenciesWithDynamicependenciesConfigurationService.java

+
+@Slf4j
+@Service
+public class DynamicDependenciesWithDynamicependenciesConfigurationService extends AbstractDynamicDependenciesService


The class name contains a typo (“Dynamicependencies”). Since this is part of a sample/public surface area, it will propagate into generated docs, logs, and code references. Rename to DynamicDependenciesWithDynamicDependenciesConfigurationService (and align related test/class references) to keep naming consistent and searchable.

Suggested change

public class DynamicDependenciesWithDynamicependenciesConfigurationService extends AbstractDynamicDependenciesService

public class DynamicDependenciesWithDynamicDependenciesConfigurationService extends AbstractDynamicDependenciesService

Copilot · 2026-02-09T17:31:05Z

...feature/loadinganalysis/withDataprepRunAnnotation/annotation/DynamicDependencySupported.java

+@Target(TYPE)
+@Retention(RUNTIME)
+@ConfigurationType("configuration")
+@Documentation("Copy/past of the annotation from tDataprepRun." +


Typo in documentation text: “Copy/past” → “Copy/paste”.

Suggested change

@Documentation("Copy/past of the annotation from tDataprepRun." +

@Documentation("Copy/paste of the annotation from tDataprepRun." +

Copilot · 2026-02-09T17:31:05Z

sample-parent/sample-features/loading-analysis/README.md

+```
+
+## Check SPI and resource loading
+The `dynamic-depdencies-with-spi` provides an input connector that will try to load:


Typo in module name: “dynamic-depdencies-with-spi” → “dynamic-dependencies-with-spi”.

Suggested change

The `dynamic-depdencies-with-spi` provides an input connector that will try to load:

The `dynamic-dependencies-with-spi` provides an input connector that will try to load:

ypiel-talend and others added 30 commits October 23, 2025 00:32

chore(QTDI-1914): Sample dynamic schema connector.

f692cad

Add 2 params

e53256c

chore(QTDI-1914): remove unused option in dso, environment info manag…

b4705a0

…ement, better clazz location.

Add 2 connectors: dataset, datastore with tests

1a76af5

Add 2 connectors: dataset, datastore with tests

8c10ae1

chore(QTDI-1914): Some adjustments.

189ac9e

chore(QTDI-1914): Working unit test.

51c2838

chore(QTDI-1914): working dynamic dependencies connectors.

a921b04

fix labels

54714b7

chore(QTDI-1914): fix artifact group id for dependency.

d52d6eb

refactor junits

e55a7a4

add working directory in output

0c8e974

chore(QTDI-1914): Improve and unit test and fix labels.

3b18693

chore(QTDI-1914): Add a connector with same behavior as datapreprun.

360f13d

fix layout for dieOnError

136b654

Fix dataprepRunAnnotation connector

c3e7ad0

feat(QTDI-1914): Cleanings.

2417f80

Add Readme

fdde448

Merge branch 'master' into ypiel/QTDI-1914_sample_connector_dyndeps

55bd365

feat(QTDI-1914): try to execute a TCK connector to check if it is wel…

0572b53

…l loaded... to continue...

Merge branch 'ypiel/QTDI-1914_sample_connector_dyndeps' of github.com…

6ff4e06

…:Talend/component-runtime into ypiel/QTDI-1914_sample_connector_dyndeps

Add function to load depended component's depends

06da286

rename

caef44a

fix junit

15c4e2d

fix(QTDI-1914): some refactoring.

132e0d9

feat(QTDI-1914): Add a connector taht uses a SPI.

578c338

feat(QTDI-1914): remove unnecessary dependency.

2dc763d

feat(QTDI-1914): Add a test on the spi.

fabc258

feat(QTDI-1914): Change the SPI interface and add a unit test.

15759e1

ypiel-talend and others added 20 commits November 28, 2025 15:31

feat(QTDI-1914): Add SPI resource for classloader customizer.

e94f6ce

feat(QTDI-1914): Rename StringMapTransformer t o StringListTransformer.

08548c9

feat(QTDI-1914): Refactoring + external dependency.

2d94660

feat(QTDI-1914): Add a class from standard dependency.

103c741

feat(QTDI-1914): Complete the readme.md.

9e50baf

feat(QTDI-1914): Complete the readme.md.

4239689

feat(QTDI-1914): fix json2Map and record builder

33dd8e2

Merge branch 'master' into ypiel/QTDI-1914_sample_connector_dyndeps

ee71c1e

# Conflicts: # sample-parent/sample-features/pom.xml

feat(QTDI-1914): Don't throw exception anymore if it misses a spi, ju…

30ac693

…st log + add an error messae in returned strings.

feat(QTDI-1914): Don't throw exception anymore if it misses a spi, ju…

ea29927

…st log + add an error messae in returned strings.

feat(QTDI-1914): Don't throw exception anymore if a resources from a …

dfe6b1f

…SPI is not loaded.

feat(QTDI-1914): Some improvements

56c3451

feat(QTDI-1914): Fix some SPI / deactivate the customer by default / …

258e6c0

…Don't fail if SPI can't be loaded.

feat(QTDI-1914): Simplify connectors forms.

a53a627

feat(QTDI-1914): Fix unit tests.

8e76aef

feat(QTDI-1914): Remove unused parameter to throw exception when a sp…

26160e5

…i in not found.

feat(QTDI-1914): A connector needs at least one parameter to be able …

ea5ea8a

…to transmit its dynamic dependencies.

feat(QTDI-1914): Add rootRepo/classpath/workingDir info in dyndepsWit…

7cbd5a1

…hSPI connector output records.

feat(QTDI-1914): Deactivate car/dependencies/scan in dynamic-dependen…

642c325

…cy-common module.

feat(QTDI-1914): Update documentation.

df43a2e

ypiel-talend requested a review from Copilot February 2, 2026 16:41

Copilot AI reviewed Feb 2, 2026

View reviewed changes

feat(QTDI-1914): Copilot code review.

ab3afe8

ypiel-talend changed the title ~~chore(QTDI-1914): Sample dynamic schema connector.~~ feat(QTDI-1914): Sample dynamic schema connector. Feb 6, 2026

ypiel-talend added 3 commits February 9, 2026 11:20

feat(QTDI-1914): Refactoring dynamic-dependencies module to loading-a…

e78dbd0

…nlysis.

feat(QTDI-1914): Fix after refactoring.

45f93fd

feat(QTDI-1914): Clean labels and icons after refactoring.

a17adc6

ypiel-talend requested a review from Copilot February 9, 2026 17:28

Copilot AI reviewed Feb 9, 2026

View reviewed changes

	"org.talend.sdk.component.sample.feature.dynamicdependencies.withspi.service.CustomizeClassLoader is enable,\n"
	"org.talend.sdk.component.sample.feature.dynamicdependencies.withspi.service.CustomizeClassLoader is enabled,\n"

-            return prop.getProperty(property);
+            final String value = prop.getProperty(property);
+            if (value == null) {
+                return "The property '%s' has not been found in resource '%s', it can't retrieve its value."
+                        .formatted(property, resource);
+            }
+            return value;


		private final Optional<S> spiImpl;

		protected AbstractSPIConsumer(final Class clazz) {


		public static final String ENTRY_RUNTIME_CLASSPATH = "runtime_classpath";

		public static final String ENTRY_WORKING_DIRECTORY = "Working_directory";

	public class DynamicDependenciesWithDynamicependenciesConfigurationService extends AbstractDynamicDependenciesService
	public class DynamicDependenciesWithDynamicDependenciesConfigurationService extends AbstractDynamicDependenciesService

	@Documentation("Copy/past of the annotation from tDataprepRun." +
	@Documentation("Copy/paste of the annotation from tDataprepRun." +

	The `dynamic-depdencies-with-spi` provides an input connector that will try to load:
	The `dynamic-dependencies-with-spi` provides an input connector that will try to load:

feat(QTDI-1914): Sample dynamic schema connector. #1114

Are you sure you want to change the base?

feat(QTDI-1914): Sample dynamic schema connector. #1114

Conversation

ypiel-talend commented Oct 22, 2025

Requirements

Why this PR is needed?

What does this PR adds (design/code thoughts)?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sonar-eks bot commented Feb 2, 2026

Quality Gate failed

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants