Why is the analytical attribute not being updated in the persist Qubole code?

Question

An event is not getting created on a change of analytical attributes, whereas the event is created for normal attributes.

val DepositAcitivityUniqueEntityDF=sqlContext.sql("select distinct(EntityId), 'No' as CheckDepositFlag from Validated_Deposit_Acitivity WHERE EntityId='10zliB3S'")
DepositAcitivityUniqueEntityDF.createOrReplaceTempView("Deposit_Acitivity_Entity_List")
DepositAcitivityUniqueEntityDF.count()
DepositAcitivityUniqueEntityDF.cache
DepositAcitivityUniqueEntityDF.show(10)

import com.reltio.analytics.data.persist._
import com.reltio.analytics.data.persist.attributes._
import com.reltio.analytics.objects.transformation._
val OrderDataPersist = af.dataPersist()
.analyticsOutput(
new AnalyticsAttributesOutputBuilder().withSkipReindex()
.withMapping(ParserMapping.fromString(CheckDepositFlagNo))
.fromDataFrame(DepositAcitivityUniqueEntityDF)
).build
.saveDataAndGetJobId()

Answer

The persist functionality just stores analytics attributes in the primary storage (Cassandra/Dynamo). After that, it launches a reindex to synchronize this new data with secondary storage (S3) as Elasticsearch.

Please remove withSkipReindex from your script (For more information regarding this, please refer to https://docs.reltio.com/riqsparksdk/datapersistapiuse.html)

In AnalyticsAttributesOutputBuilder(), you can:

• Skip launching re-index: use method .withSkipReindex().
  • Example: new AnalyticsAttributesOutputBuilder().withSkipReindex().
  • As a result, reindexing will be skipped and the job payload will contain "reindex":false.