Conversation
sumitjnn
reviewed
Apr 5, 2023
Comment on lines
19
to
44
| # Scenario: Sanity test from Oracle to Big Query | ||
| # Given Open DataFusion Project with replication to configure pipeline | ||
| # When Enter input plugin property: "name" with value: "pipelineName" | ||
| # And Click on the Next button | ||
| # And Select Oracle as Source | ||
| # Then Replace input plugin property: "host" with value: "host" for Credentials and Authorization related fields | ||
| # Then Replace input plugin property: "port" with value: "port" for Credentials and Authorization related fields | ||
| # Then Click plugin property: "region" | ||
| # Then Click plugin property: "regionOption" | ||
| # Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields | ||
| # Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields | ||
| # Then Replace input plugin property: "sid" with value: "database" for Credentials and Authorization related fields | ||
| # Then Click on the Next button | ||
| # Then Replace input plugin property: "loadInterval" with value: "loadInterval" | ||
| # Then Click on the Next button | ||
| # Then Validate Table is available and can be selected "ABC.E2E-sanity" | ||
| # And Click on the Next button | ||
| # And Click on the Next button | ||
| # And Click on the Next button | ||
| # Then Deploy the replication pipeline | ||
| # And Run the replication Pipeline | ||
| # Then Open the logs | ||
| # And Wait till pipeline is in running state and check if no errors occurred | ||
| # Then Verify expected Oracle records in target BigQuery table | ||
| # And Capture raw logs | ||
| # Then Close the pipeline logs and stop the pipeline |
Contributor
There was a problem hiding this comment.
Will this comment block be removed?
| # And Capture raw logs | ||
| # Then Close the pipeline logs and stop the pipeline | ||
|
|
||
| # @ORACLE_SOURCE |
Contributor
There was a problem hiding this comment.
What's the significance of this annotation? Why is it commented out?
| # Then Close the pipeline logs and stop the pipeline | ||
|
|
||
| # @ORACLE_SOURCE | ||
| Scenario: To verify snapshot and cdc from Oracle to Big Query successfully |
Contributor
There was a problem hiding this comment.
Suggested change
| Scenario: To verify snapshot and cdc from Oracle to Big Query successfully | |
| Scenario: To verify replication of snapshot and cdc data from Oracle to Big Query successfully |
Comment on lines
33
to
42
| datatypeColumnsForSanityTesting=('USER1', 'M','ABCDEF','ABC','ABC','�','� 好�?�?','� 好�?�?','AAAAaoAATAAABrXAAA',1234,1234.56789,\ | ||
| 1234.56789,1234.56789,1234.56789,1234.56789,1234.56789,1234.56789,1234.5679,\ | ||
| 1234.56789,1234.5679,1234.5679,1234.56789,TIMESTAMP'2023-01-01 2:00:00',TIMESTAMP'2023-01-01 2:00:00',\ | ||
| TIMESTAMP '2023-01-01 00:00:00.000000') | ||
|
|
||
| datatypeValuesForSanityTesting=('USER1', 'M','ABCDEF','ABC','ABC','�','� 好�?�?','� 好�?�?',\ | ||
| 'AAAAaoAATAAABrXAAA',1234,1234.56789,\ | ||
| 1234.56789,1234.56789,1234.56789,1234.56789,1234.56789,1234.56789,1234.5679,\ | ||
| 1234.56789,1234.5679,1234.5679,1234.56789,TIMESTAMP'2023-01-01 2:00:00',TIMESTAMP'2023-01-01 2:00:00',\ | ||
| TIMESTAMP '2023-01-01 00:00:00.000000') No newline at end of file |
Contributor
There was a problem hiding this comment.
What's the reason for storing these in property file? Are we storing rows and types like this for existing e2e tests?
Contributor
Author
There was a problem hiding this comment.
Yes.For connectors, it is done this way.
|
|
||
| public static void waitTillPipelineIsRunningAndCheckForErrors() throws InterruptedException { | ||
| //wait for datastream to startup | ||
| int defaultTimeout = Integer.parseInt(PluginPropertyUtils.pluginProp("datastream.timeout")); |
Contributor
There was a problem hiding this comment.
Rename it to more general term like pipeline initialization time instead of datastream.timeout
| bqRow.remove("_is_deleted"); | ||
| bqRow.remove("_sequence_num"); | ||
|
|
||
| ValidationHelper.compareBothResponses( bqRow, oracleRecord); |
Contributor
There was a problem hiding this comment.
Suggested change
| ValidationHelper.compareBothResponses( bqRow, oracleRecord); | |
| compareRecords( bqRow, oracleRecord); |
| if (port!= null && !port.isEmpty()) { | ||
| PluginPropertyUtils.addPluginProp("port", port); | ||
| } | ||
| String sapHost = System.getenv("ORACLE_HOST"); |
| @Before(order = 1, value = "@ORACLE_SOURCE") | ||
| public static void overridePropertiesFromEnvVarsIfProvided() { | ||
| String username = System.getenv("ORACLE_USERNAME"); | ||
| if (username != null && !username.isEmpty()) { |
Contributor
There was a problem hiding this comment.
what happens if property is null?
Contributor
Author
There was a problem hiding this comment.
test will not run like username is used for creating tables, that will fail thereby ending tests
|
|
||
| @Before(order = 4, value = "@ORACLE_SOURCE") | ||
| public static void getOracleRecordsAsMap() throws SQLException, ClassNotFoundException { | ||
| sourceOracleRecords = OracleClient.getOracleRecordsAsMap(tableName, schemaName); |
Contributor
There was a problem hiding this comment.
This is for logging expected records before test starts?
|
|
||
| @After(order = 1, value = "@ORACLE_SOURCE_TEMP") | ||
| public static void dropTables() throws SQLException, ClassNotFoundException { | ||
| OracleClient.deleteTables(schemaName, tableName); |
Contributor
There was a problem hiding this comment.
Should we drop BQ table as well?
Contributor
Author
There was a problem hiding this comment.
Yes we can. But having target table is good for validation after e2e tests are done
sumitjnn
reviewed
Apr 10, 2023
| @@ -0,0 +1,50 @@ | |||
| # | |||
Contributor
There was a problem hiding this comment.
I think it would be better to rename the file to SanityTests.feature to reflect the intent
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.