Regression testing
This section describes what you need to know about regression testing in iCore Cloud Services. If you need further assistance, please contact us at support@icore.se.
What is a regression test?
Regression testing means re-running functional and non-functional tests to ensure that previously developed and tested integrations still perform as expected after a change. If they don't, it would be referred to as a regression.
Regression tests will be run when:
- A new deployment package is deployed, before a change is committed to your Production environment.
- Any major changes are made to the iCCS platform, for example when iCore Integration Suite is updated.
Deploying a new regression test
To deploy a new regression test, submit a Regression test request in the iCore Self-Service Portal. You can also send an email to support@icore.se.
The request must contain the following files:
Regression tests management file
The Regression test management file is an XML file that keeps track of all your regression tests. This file contains:
- Customer: Your iCCS customer code.
- TestCases: A list of all your current regression tests.
- id: An ID used by the regressions test manager, it can be any ID.
- Configuration file: the relative search path for the configuration file used by this regression test.
- TestSets: Any number of test sets can be created to manage how the regression tests will be run.
- TestSet: A container used to manage how and when different regression tests will run.
- name: a name used by the regression test manager to identify the current test set.
- type: Can be set to "Sync" or "Async", depending on if the test cases in this test set should run in sequence or in parallel.
- ActiveInAll: Can be set to "Yes" or "No". If set to Yes, all tests in this test set will be run if no test set or test case is defined when running your regression tests.
- TestCaseRef id: a reference to the test case defined under "TestCases".
Management file example
<?xml version="1.0" encoding="UTF-8"?>
<RegressionTestManager xmlns="icorepaas.com/regressiontests/custtestmanager/config/1.0">
<Customer code="e10">
<TestCases>
<TestCase id="e10_ INT033_01">INT033_01 OrderSync\e10_ INT033_01.xml</TestCase>
<TestCase id="e10_ INT034_01">INT034_01 OrderDeliveryReport\e10_ INT034_01.xml</TestCase>
<TestCase id="e10_ INT035_01">INT035_01 DispatchSync\e10_ INT035_01.xml</TestCase>
<TestCase id="e10_ INT036_01">INT036_01 DispatchContentReceive\e10_ INT036_01.xml</TestCase>
<TestCase id="e10_ INT037_01">INT037_01 DispatchReceiveReport\e10_ INT037_01.xml</TestCase>
<TestCase id="e10_ INT038_01">INT038_01 BalanceList\e10_ INT038_01.xml</TestCase>
<TestCase id="e10_ INT039_01">INT039_01 BalanceChange\e10_ INT039_01.xml</TestCase>
<TestCase id="e10_ INT040_01">INT040_01 ConfirmBOD\e10_ INT040_01.xml</TestCase>
</TestCases>
<TestSets>
<TestSet name="Group1" type="Sync" activeInAll="Yes">
<TestCaseRef id="e10_ INT033_01"/>
<TestCaseRef id="e10_ INT034_01"/>
<TestCaseRef id="e10_ INT035_01"/>
<TestCaseRef id="e10_ INT036_01"/>
<TestCaseRef id="e10_ INT037_01"/>
<TestCaseRef id="e10_ INT038_01"/>
<TestCaseRef id="e10_ INT039_01"/>
<TestCaseRef id="e10_ INT040_01"/>
</TestSet>
<TestSet name="Group2" type="Async" activeInAll="No">
<TestCaseRef id="e10_ INT001_01"/>
<TestCaseRef id="e10_ INT007_01"/>
<TestCaseRef id="e10_ INT002_01"/>
</TestSet>
</TestSets>
</Customer>
</RegressionTestManager>
Configuration file
Attach the endpoint configuration file for the regression test.
Configuration file example
This example has EndpointType set to FILE both in InData and OutData Config.
<?xml version="1.0" encoding="UTF-8"?>
<RegressionTest xmlns="icorepaas.com/regressiontests/testcase/config/1.0">
<TestCase version="1.0">
<CustomerCode>eXX</CustomerCode>
<Id>ISC00X_01</Id>
<Description>Demo Test-Case for customer, integration scenario ISC00X Test 01</Description>
<InputDataFilename>.\eXX_ ISC00X_01 InputData.xml</InputDataFilename>
<ResultDataFilename>.\eXX_ ISC00X_01 ResultData.xml</ResultDataFilename>
<!-- FilterType can be "xml" / "text" / "none" -->
<FilterFilename FilterType="xml">.\eXX_ ISC00X_01 FilterConfig.xml</FilterFilename>
<!-- Timeout is in TimeSpan-format. HH:MM:SS -->
<Timeout>00:30:00</Timeout>
<!-- All parameter referes to GFC! -->
<InDataConfig EndpointType="FILE">
<Server>
<Server_S_FTP_S>
<Connection_Timeout>60000</Connection_Timeout><!-- 60000 Timeout (in milliseconds) -->
<Encrypt_Data_Channel>False</Encrypt_Data_Channel>
<FTPCommand></FTPCommand><!-- Command sent directly after the connection has been established. -->
<FTPCommandAcceptedErrorCodes></FTPCommandAcceptedErrorCodes><!-- List of error codes (separated with commas). -->
<FTPPassword></FTPPassword>
<FTPPort>21</FTPPort><!-- 21 Port -->
<FTPServer></FTPServer><!-- Server -->
<FTPUser></FTPUser><!-- User -->
<FTPType>FTP</FTPType><!-- FTP FTP / FTPS / SFTP -->
<Ignore_error_codes></Ignore_error_codes><!-- Error codes separated by comma (e.g. 123, 234, 345) -->
<List_Is_Case_Sensitive>False</List_Is_Case_Sensitive><!-- True / False -->
<List_method>Default</List_method><!-- Default / MLSD -->
<Passive_FTP_mode>True</Passive_FTP_mode><!-- True / False -->
<Remote_codepage>20127</Remote_codepage><!-- Default is 20127 (US-ASCII) -->
<Server_directory_separator>/</Server_directory_separator>
<SSL_Auth_Command>Auto</SSL_Auth_Command>
<SSL_Certificate></SSL_Certificate>
<SSL_Enable_Validation>False</SSL_Enable_Validation>
<SSL_Mode>Explicit</SSL_Mode><!-- Implicit / Explicit -->
<SSL_TLS_Version>All</SSL_TLS_Version><!-- All, Ssl2, Ssl3, Tls1, Tls11,Tls12 -->
<SSL_Trusted_Storage></SSL_Trusted_Storage>
<SSL_Use>False</SSL_Use><!-- True / False -->
<Transfer_type>Binary</Transfer_type><!-- Binary / ASCII -->
</Server_S_FTP_S>
<Server_SFTP_Security>
<FILE_Download_block_size>0</FILE_Download_block_size>
<FILE_Force_compression>False</FILE_Force_compression>
<FILE_Upload_block_size>0</FILE_Upload_block_size>
<HTTP_Authentication_method>NoAuthentication</HTTP_Authentication_method><!-- Basic, Digest, NoAuthentication, NTLM -->
<HTTP_proxy_password></HTTP_proxy_password>
<HTTP_proxy_port>0</HTTP_proxy_port>
<HTTP_proxy_server></HTTP_proxy_server>
<HTTP_proxy_username></HTTP_proxy_username>
<HTTP_Use_proxy>False</HTTP_Use_proxy>
<Implicit_interactive_logon>False</Implicit_interactive_logon>
<Interactive_logon></Interactive_logon>
<Private_key_file></Private_key_file>
<Private_key_password></Private_key_password>
<Server_validation>None</Server_validation><!-- None, Public_Key -->
<Servers_public_key_file></Servers_public_key_file>
<Socket_timeout>0</Socket_timeout>
<SOCKS_Enable_authentication>False</SOCKS_Enable_authentication>
<SOCKS_IPv6>False</SOCKS_IPv6>
<SOCKS_password></SOCKS_password>
<SOCKS_port>1080</SOCKS_port>
<SOCKS_Resolve_address_on_server>False</SOCKS_Resolve_address_on_server>
<SOCKS_server></SOCKS_server>
<SOCKS_Use>False</SOCKS_Use>
<SOCKS_username></SOCKS_username>
<SOCKS_version>Version5</SOCKS_version><!-- Version4, Version5 -->
</Server_SFTP_Security>
</Server>
<SendEndpoint>
<Endpoint_File_Send>
<Destination_FileName></Destination_FileName><!-- NOTE: No variables allowed! -->
<Destination_Folder></Destination_Folder><!-- NOTE: No variables allowed! -->
<Destination_FileName_Upload_Extension>.uploading</Destination_FileName_Upload_Extension><!-- Destination FileName extension during upload. -->
<Overwrite_if_File_exists>N</Overwrite_if_File_exists><!-- Y / N, Overwrite if File exists in the Destination Folder. -->
</Endpoint_File_Send>
<Endpoint_S_FTP_S_Send>
<Destination_FileName></Destination_FileName><!-- NOTE: No variables allowed! -->
<Destination_Folder></Destination_Folder><!-- NOTE: No variables allowed! -->
<Destination_FileName_Upload_Extension>.uploading</Destination_FileName_Upload_Extension><!-- Destination FileName extension during upload. -->
<OverwriteMode>Skip</OverwriteMode><!-- Overwrite, Skip, Append, Error or None -->
</Endpoint_S_FTP_S_Send>
</SendEndpoint>
</InDataConfig>
<OutDataConfig EndpointType="FILE">
<Server>
<Server_S_FTP_S>
<Connection_Timeout>60000</Connection_Timeout><!-- 60000 Timeout (in milliseconds) -->
<Encrypt_Data_Channel>False</Encrypt_Data_Channel>
<FTPCommand></FTPCommand><!-- Command sent directly after the connection has been established. -->
<FTPCommandAcceptedErrorCodes></FTPCommandAcceptedErrorCodes><!-- List of error codes (separated with commas). -->
<FTPPassword></FTPPassword>
<FTPPort>21</FTPPort><!-- 21 Port -->
<FTPServer></FTPServer><!-- Server -->
<FTPUser></FTPUser><!-- User -->
<FTPType>FTP</FTPType><!-- FTP FTP / FTPS / SFTP -->
<Ignore_error_codes></Ignore_error_codes><!-- Error codes separated by comma (e.g. 123, 234, 345) -->
<List_Is_Case_Sensitive>False</List_Is_Case_Sensitive><!-- True / False -->
<List_method>Default</List_method><!-- Default / MLSD -->
<Passive_FTP_mode>True</Passive_FTP_mode><!-- True / False -->
<Remote_codepage>20127</Remote_codepage><!-- Default is 20127 (US-ASCII) -->
<Server_directory_separator>/</Server_directory_separator>
<SSL_Auth_Command>Auto</SSL_Auth_Command>
<SSL_Certificate></SSL_Certificate>
<SSL_Enable_Validation>False</SSL_Enable_Validation>
<SSL_Mode>Explicit</SSL_Mode><!-- Implicit / Explicit -->
<SSL_TLS_Version>All</SSL_TLS_Version><!-- All, Ssl2, Ssl3, Tls1, Tls11,Tls12 -->
<SSL_Trusted_Storage></SSL_Trusted_Storage>
<SSL_Use>False</SSL_Use><!-- True / False -->
<Transfer_type>Binary</Transfer_type><!-- Binary / ASCII -->
</Server_S_FTP_S>
<Server_SFTP_Security>
<FILE_Download_block_size>0</FILE_Download_block_size>
<FILE_Force_compression>False</FILE_Force_compression>
<FILE_Upload_block_size>0</FILE_Upload_block_size>
<HTTP_Authentication_method>NoAuthentication</HTTP_Authentication_method><!-- Basic, Digest, NoAuthentication, NTLM -->
<HTTP_proxy_password></HTTP_proxy_password>
<HTTP_proxy_port>0</HTTP_proxy_port>
<HTTP_proxy_server></HTTP_proxy_server>
<HTTP_proxy_username></HTTP_proxy_username>
<HTTP_Use_proxy>False</HTTP_Use_proxy>
<Implicit_interactive_logon>False</Implicit_interactive_logon>
<Interactive_logon></Interactive_logon>
<Private_key_file></Private_key_file>
<Private_key_password></Private_key_password>
<Server_validation>None</Server_validation><!-- None, Public_Key -->
<Servers_public_key_file></Servers_public_key_file>
<Socket_timeout>0</Socket_timeout>
<SOCKS_Enable_authentication>False</SOCKS_Enable_authentication>
<SOCKS_IPv6>False</SOCKS_IPv6>
<SOCKS_password></SOCKS_password>
<SOCKS_port>1080</SOCKS_port>
<SOCKS_Resolve_address_on_server>False</SOCKS_Resolve_address_on_server>
<SOCKS_server></SOCKS_server>
<SOCKS_Use>False</SOCKS_Use>
<SOCKS_username></SOCKS_username>
<SOCKS_version>Version5</SOCKS_version><!-- Version4, Version5 -->
</Server_SFTP_Security>
</Server>
<ReceiveEndpoint>
<Endpoint_File_Receive>
<Delay_Process>0</Delay_Process><!-- Time in milliseconds before the file is picked up. -->
<Delete_File_After_Import>Y</Delete_File_After_Import><!-- Y / N -->
<Settle_Time>0</Settle_Time><!-- Time in milliseconds that the file has to be unchanged (filesize) before iCore picks it up. Applies to each individually found file. -->
<Settle_Time_Timeout>00:00:30</Settle_Time_Timeout><!-- TimeSpan (hh:mm:ss) -->
<Source_File_Filter>*.*</Source_File_Filter><!-- Source File Filter (Search Pattern) -->
<Source_File_List_Order>CreationTime</Source_File_List_Order><!-- CreationTime, Name, Extension -->
<Source_File_List_Order_Type>ASC</Source_File_List_Order_Type><!-- ASC=Ascending, DESC=Descending -->
<Source_Folder></Source_Folder>
<Source_Move_Folder></Source_Move_Folder><!-- Move File after Processing -->
</Endpoint_File_Receive>
<Endpoint_S_FTP_S_Receive>
<Delay_Process>0</Delay_Process><!-- Time in milliseconds before the file is picked up. -->
<Delete_File_After_Import>Y</Delete_File_After_Import><!-- Y / N -->
<Settle_Time>0</Settle_Time><!-- Time in milliseconds that the file has to be unchanged (filesize) before iCore picks it up. Applies to each individually found file. -->
<Settle_Time_Timeout>00:00:30</Settle_Time_Timeout><!-- TimeSpan (hh:mm:ss) -->
<Source_File_Filter>*.*</Source_File_Filter><!-- Source File Filter (Regular Expression) -->
<Source_Folder></Source_Folder>
<Source_List_Filter></Source_List_Filter>
<Source_List_Filter_Options>FilterOnServer</Source_List_Filter_Options><!-- FilterOnServer, FilterOnClient, FilterOnClientCaseSensitive -->
<Source_Move_Folder></Source_Move_Folder><!-- Move File after Processing -->
</Endpoint_S_FTP_S_Receive>
</ReceiveEndpoint>
</OutDataConfig>
</TestCase>
</RegressionTest>
When EndpointType in IndataConfig is set to API, the file looks a little different, and the OutDataConfig section does not have any meaning here. See example below:
<?xml version="1.0" encoding="UTF-8"?>
<RegressionTest xmlns="icorepaas.com/regressiontests/testcase/config/1.0">
<TestCase version="1.0">
<CustomerCode>eXX</CustomerCode>
<Id>ISC00X_01</Id>
<Description>Demo Test-Case for customer, integration scenario ISC00X Test 01</Description>
<InputDataFilename>.\eXX_ ISC00X_01 InputData.json</InputDataFilename>
<ResultDataFilename>.\eXX_ ISC00X_01 ResultData.xml</ResultDataFilename>
<!-- FilterType can be "xml" / "text" / "none" -->
<FilterFilename FilterType="xml">.\eXX_ ISC00X_01 FilterConfig.xml</FilterFilename>
<!-- Timeout is in TimeSpan-format. HH:MM:SS -->
<Timeout>00:30:00</Timeout>
<!-- All parameter referes to GFC! -->
<InDataConfig EndpointType="API"><!--If this is set to API, OutDataConfig EndpointType should also be set to API. -->
<Server/>
<SendEndpoint>
<Endpoint_API>
<Destination_Host>https://eXX.icorepaas.com/eXX/api/icore/V1/OrderTrigger</Destination_Host>
<!-- IF you have any request headers, please state them here -->
<Destination_Http_Headers>
<!-- You can send more than one Http Header, create a new Destination_Http_Header tag here below -->
<Destination_Http_Header>
<Name>SomeHeaderName</Name>
<Value>SomeHeaderValue</Value>
</Destination_Http_Header>
</Destination_Http_Headers>
<ContentType>application/json</ContentType><!-- Specify content-type for the content. If not specified, application/json will be used. -->
<!-- Here we need to choose one of the follow security, if there is none, please leave empty -->
<Security>
<Basic>
<Username>username</Username>
<Password>abc123</Password>
</Basic>
<OAuth2>
<ValuePrefix></ValuePrefix><!--When using OAuth2, this should probably always be set to 'Bearer'.-->
<AccessToken></AccessToken>
</OAuth2>
<ApiKey>
<Key></Key>
</ApiKey>
<Custom>
<Name></Name>
<Value></Value>
</Custom>
</Security>
</Endpoint_API>
</SendEndpoint>
</InDataConfig>
<OutDataConfig EndpointType="API"><!--If InDataConfig EndpointType is set to API, this should also be set to API. -->
<Server/>
<ReceiveEndpoint>
<Endpoint_File_Receive>
<Delay_Process></Delay_Process>
<Delete_File_After_Import></Delete_File_After_Import>
<Settle_Time></Settle_Time>
<Settle_Time_Timeout></Settle_Time_Timeout>
<Source_File_Filter></Source_File_Filter>
<Source_File_List_Order></Source_File_List_Order>
<Source_File_List_Order_Type></Source_File_List_Order_Type>
<Source_Folder></Source_Folder>
<Source_Move_Folder/>
</Endpoint_File_Receive>
</ReceiveEndpoint>
</OutDataConfig>
</TestCase>
</RegressionTest>
Input data file
Attach the input data file, which will be used to start the regression test. In cases when Endpoint Type is set to API, this file is expected to contain the content for the API call. The same file will be used every time the regression test is run.
Result data file
Attach the result data file. This file represents the desired result from running the input file through your integration. The result from the test run will be compared to this file to confirm that no breaking changes have been made.
Filter file
Attach the filter file. This file is used to filter out dynamic data that changes between runs. Without this filter, the result will not match your provided result file. Depending on whether the result file is a text file or an XML file, you can see example files below.
Filter example: text file
<?xml version="1.0" encoding="UTF-8"?>
<StripConfig xmlns="icorepaas.com/regressiontests/util/strip/regex/1.0">
<StripRow RowNo="1">
<RemoveChar Position="54" Count="17"/>
<RemoveChar Position="84" Count="5"/>
<RemoveChar Position="4903" Count="5"/>
<RemoveChar Position="4915" Count="5"/>
</StripRow>
<StripRow RowNo="10">
<RemoveChar Position="4900" Count="17"/>
</StripRow>
</StripConfig>
Filter example: XML file
<ReplaceConfig xmlns="icorepaas.com/regressiontests/util/replace/xpath/1.0">
<Actions>
<ReplaceElementValues Name="Rule 1">
<XPathFilter>//ns1:Order/ns1:OrderHead/ns1:OrderDate</XPathFilter>
<NamespaceDecl>xmlns:ns1="iCCS/ReplaceXPathConfig/sample/1.0"</NamespaceDecl>
<NewElementValue SetNewValue="true">2020-01-05</NewElementValue>
<SetAttibutes/>
</ReplaceElementValues>
<ReplaceElementValues Name="Rule 2">
<XPathFilter>//ns1:Order/ns1:OrderLines/ns1:OrderLine[position()=2]</XPathFilter>
<NamespaceDecl>xmlns:ns1="iCCS/ReplaceXPathConfig/sample/1.0"</NamespaceDecl>
<NewElementValue SetNewValue="false"></NewElementValue>
<SetAttibutes>
<Attibute Name="No" NewValue="X"/>
<Attibute Name="Qty" NewValue="88"/>
</SetAttibutes>
</ReplaceElementValues>
</Actions>
</ReplaceConfig>
Filter schema: Text file
<?xml version="1.0" encoding="utf-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns="icorepaas.com/regressiontests/util/strip/regex/1.0" xmlns:ns1="icorepaas.com/regressiontests/util/strip/regex/1.0" targetNamespace="icorepaas.com/regressiontests/util/strip/regex/1.0" elementFormDefault="qualified">
<!--
NOTE!
The order of StripRow must be given in RowNo order.
The order of RemoveChar must be given in Position order.
-->
<xs:element name="StripConfig">
<xs:complexType>
<xs:sequence>
<xs:element ref="StripRow" maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="StripRow">
<xs:complexType>
<xs:sequence>
<xs:element ref="RemoveChar" maxOccurs="unbounded"/>
</xs:sequence>
<xs:attribute name="RowNo" type="xs:string" use="required"/>
</xs:complexType>
</xs:element>
<xs:element name="RemoveChar">
<xs:complexType>
<xs:attribute name="Position" type="xs:string" use="required"/>
<xs:attribute name="Count" type="xs:string" use="required"/>
</xs:complexType>
</xs:element>
</xs:schema>
Filter schema: XML file
<?xml version="1.0" encoding="utf-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns="icorepaas.com/regressiontests/util/replace/xpath/1.0" xmlns:ns1="icorepaas.com/regressiontests/util/replace/xpath/1.0" targetNamespace="icorepaas.com/regressiontests/util/replace/xpath/1.0" elementFormDefault="qualified">
<xs:element name="ReplaceConfig">
<xs:complexType>
<xs:sequence>
<xs:element ref="Actions" maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="Actions">
<xs:complexType>
<xs:sequence>
<xs:element ref="ReplaceElementValues" minOccurs="0" maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="ReplaceElementValues">
<xs:complexType>
<xs:sequence>
<xs:element ref="XPathFilter"/>
<xs:element ref="NamespaceDecl"/>
<xs:element ref="NewElementValue"/>
<xs:element ref="SetAttibutes"/>
</xs:sequence>
<xs:attribute name="Name" type="xs:string" use="required"/>
</xs:complexType>
</xs:element>
<xs:element name="XPathFilter">
<xs:complexType/>
</xs:element>
<xs:element name="NamespaceDecl">
<xs:complexType/>
</xs:element>
<xs:element name="NewElementValue">
<xs:complexType>
<xs:attribute name="SetNewValue" type="xs:string" use="required"/>
</xs:complexType>
</xs:element>
<xs:element name="SetAttibutes">
<xs:complexType>
<xs:sequence>
<xs:element ref="Attibute" maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="Attibute">
<xs:complexType>
<xs:attribute name="Name" type="xs:string" use="required"/>
<xs:attribute name="NewValue" type="xs:string" use="required"/>
</xs:complexType>
</xs:element>
</xs:schema>
Running regression tests
To run a regression test, please contact support@icore.se.
note
Only iCore personnel can run regression tests. Please contact support@icore.se for more information.
Are regression tests mandatory?
No, but we recommend using regression tests to make sure your integrations continue to run uninterrupted when changes are made to your iCore system as well as the iCore Cloud Services platform.