Most Recent Record Processed

In TIBCO Cloud™ Integration - Connect, Net Change refers to new and updated records in a source datastore. Most Recent Record Processed is the datetime value used by TIBCO Cloud™ Integration - Connect to determine which records in a source datastore are new or have been updated.

If this option is enabled, each time an app executes, TIBCO Cloud™ Integration - Connect compares the Most Recent Record Processed value to a datetime value in each Source record. If the datetime value in the Source is newer than the Most Recent Record Processed value, the Source record is selected by the source query and processed by the app. Processing only new or updated records reduces the total number of records processed by TIBCO Cloud™ Integration - Connect.

When a datetime is configured on the Query block on the Block Properties Net Change Tab to query for new and updated records, that configuration is treated as an additional filter. The Net Change datetime filter is applied as an AND after any other filters specified on the Block Properties Filter Tab. TIBCO Cloud™ Integration - Connect builds a query combining both the Net Change filter and the filters on the Filter tab. See Net Change And Filters for an example.

Some Connectors for TIBCO Cloud™ Integration - Connect only support one filter. For those Connectors you can use either Net Change or one filter on the Filter tab, not both.

Date Vs. DateTime

If an entity only has a date field with no time being used for Net Change, some records might be reprocessed depending on when the app executes. In that case, using the date only field to query for new and updated records returns all records starting at 12:00 am of the date in the Most Recent Record Processed field through the time that the app executes.

For example, if the value in Most Recent Record Processed is June 10, 2017 and you run the app at 1:00 pm on June 12, 2017, the query returns all records created or updated from 12:00 am on June 10, 2017 through June 12, 2017 at 1:00 pm. If the last new or modified record had a date value of June 12, 2017 and you run the app again at 4:00 pm on June 12, 2017, the query returns all records created or updated from 12:00 on June 12, 2017 through 4:00 pm. Records created between 12:00 am and 1:00 pm on June 12, 2017 are reprocessed.

Best practice is to execute apps by querying entities with date only fields at 12:00 am to minimize the number of records that are reprocessed.

Last Modified Date Field

TIBCO Cloud™ Integration - Connect has a Last Modified Date concept that refers to the last time a Source record was changed. The Last Modified Date value is compared to the Most Recent Record Processed value by the query to locate only new and updated Source records. The value of the Last Modified Date field is determined as follows:

• Data replication apps — Connectors that support Data replication apps specify the Source field to be used as the Last Modified Date field for each entity included in the app. When the Data replication app executes it retrieves the datetime value for each Source record from the field specified by the Connector. If the entity being processed does not have a datetime field that can be used as the Last Modified Date field, then all of the records for that entity are processed each time the app runs. To reduce the number of records processed, configure the Data replication app to use the Recommended Entities or Selected Entities instead of All Entities.
• On schedule apps — When the Process only records created or updated since last run option is enabled in a flow, you are required to select a datetime field in the Source entity to serve as the Last Modified Date field. The value of the selected field is compared against the Most Recent Record Processed value each time the app is executed. Be sure to select a datetime field that is updated any time a Source record is modified.

Most Recent Record Processed Value

The Most Recent Record Processed value is determined as follows: 

• Data replication apps — When TIBCO Cloud™ Integration - Connect replicates data into a target datastore, it filters source data for each table based on the Last Modified Date field specified by the Connector. TIBCO Cloud™ Integration - Connect determines which field is the Last Modified Date field for a particular table, and reviews the dates in that field in the target datastore to locate the most recent datetime. That value becomes the Most Recent Record Processed value for the selected table in the current execution of the Data replication app. If there is a Fatal Exception during replication, the next time the Data replication app executes TIBCO Cloud™ Integration - Connect subtracts 500 milliseconds from the date/time field found in the target table where the failure occurred. This ensures that no records are missed, however, a few records may be reprocessed unnecessarily.
• On schedule apps — When TIBCO Cloud™ Integration - Connect finishes executing an On schedule app, it stores the datetime value of the Source record with the most recent Last Modified Date datetime. The value of the Last Modified Date field for that record becomes the Most Recent Record Processed value for the next execution of the app. The Most Recent Record Processed value is stored with the app and is updated each time the app executes.

  A Source record triggers an update to the Most Recent Record Processed date, if it was used as the source in an operation block that modifies records in the target, including: Create, Delete, Update, Update/Insert, or Upsert.

  A Source record does not trigger an update to the Most recent record processed date, if:

  • The operation does not write any information from the source record into the target system.
  • It was used by an operation block that does not modify the target, such as Fetch or Lookup.
  Note: For On schedule apps, the Most Recent Record Processed value can be manually reset to retrieve older records, if necessary. See Query Block and Block Properties Net Change Tab.

Using Most Recent Record Processed

To take advantage of the Most Recent Record Processed value, the Process only records created or updated since last run option must be enabled.

• Data replication apps — The Process only records created or updated since last run option is enabled internally and cannot be disabled.
• On schedule apps — The Process only records created or updated since last run option is enabled in the Query Block in a flow. See Block Properties Net Change Tab.

Related Topics

Query Block

Managing Failed Data Replication App Execution History Records